How Sigilix reports recall, precision, F1, and false positives with the fixture boundaries visible.
A review product has to compete on belief. It is not enough to post more comments or catch a bug once in a demo. The result has to survive audit: what was caught, what was missed, what was a false alarm, and which comparison is actually fair.
That is why Sigilix keeps benchmark evidence close to its source. The figure below separates recall, precision, F1, false positives, and severity recall instead of flattening them into one blended claim.
Benchmarks with the context kept intact.
The Sigilix numbers come from our own e2e bake-off fixtures. The peer numbers are vendor-published figures on their own data — not the same fixture, shown for context, not as a head-to-head result. Each metric opens into a native figure, with severity splits and ranges kept separate so the numbers stay easy to read.
The same comparison split by bug severity. Each mini-chart keeps the peer set together so the distribution is readable at a glance.
Critical
- Sigilix
- Greptile
- Bugbot
- Copilot
- CodeRabbit
- Graphite
High
- Sigilix
- Greptile
- Bugbot
- Copilot
- CodeRabbit
- Graphite
Medium + low
- Sigilix
- Greptile
- Bugbot
- Copilot
- CodeRabbit
- Graphite
How to read this: the Sigilix figures are measured on Sigilix's own e2e bake-off fixtures. The peer figures are vendor-published on their own data, so this is not a same-fixture, apples-to-apples comparison — it shows how Sigilix performs on its own fixtures alongside published peer numbers for context, not a head-to-head win.
How to read the fixture.
The Sigilix column is measured on a hand-built, twice-verified bug fixture, scored with K-repeated runs and a human audit of unmatched findings.
The goal is not a best draw. It is a repeatable view of whether the reviewer can catch real issues without flooding the engineer with noise.
Why the peer numbers stay bounded.
The peer columns are vendors' published figures or independent Martian tracks. They are useful context, but they are not presented as a same-fixture comparison.
Ranges stay as ranges. Missing values stay missing. That keeps the comparison readable without pretending the public data is cleaner than it is.
What the benchmark is for.
The numbers guide the product work: raise recall without weakening precision, reduce false positives by proof, and keep the merge-blocking tier honest.
Benchmarks are useful only if they make the next engineering decision clearer.
We built the benchmark backwards from trust.
A catch only counts when it preserves the engineer's reasoning trail: where the failure lives, why it matters, how severe it is, what the reviewer said, and which line-level evidence made the finding real. The set uses production-shaped failures: auth flows, audit logs, queue semantics, incident rules, metrics, and configuration edges.
Audit-log pagination skips records
The catch has to follow the cursor through filtering and show how a high-volume audit view can silently omit records a compliance review expects to see.
OAuth callback accepts missing state
The reviewer has to identify the unverified callback state and explain how the login flow can bind the wrong session or fail open.
Bulk-delete reports success on failure
The finding counts only if it follows the changed error contract to the UI or caller that now treats an incomplete destructive operation as successful.
Queue shutdown loses in-flight work
The review has to connect the exception handling path to jobs that are acknowledged before the worker can safely persist or retry them.
Incident rule reads stale config
The catch must trace the updated setting into the detector path and show where the old value still decides whether an incident is created.
Metric tags split the same shard
The reviewer has to point out that two tag names describe the same dimension, making dashboards and alerts undercount the affected shard.
Zero sampling rate is ignored
The catch has to show that an intentional 0.0 configuration is treated as missing, turning off the user's explicit sampling behavior.
| Recall, all planted bugs | 84% |
|---|---|
| Precision, hand-audited | 91% |
| False-positive rate, lower is better | 9% |
Where these numbers help, and where they do not
Where this helps
- Recall, precision, F1, and false positives are reported separately, so you can see what was caught, what was missed, and what was a false alarm instead of one blended score.
- The fixture is scored with repeated runs and a human audit of unmatched findings, so the result is a repeatable view rather than a best single draw.
- Findings carry the evidence that made them real, which is what lets a review become repair work instead of a comment the team has to re-litigate.
Where it does not
- The Sigilix column is measured on our own hand-built fixture. It is not a same-fixture, apples-to-apples comparison against other vendors.
- The peer figures are vendor-published on their own data or independent Martian tracks. They are context, not a head-to-head result, which is why ranges stay ranges and missing values stay missing.
- A fixture is not your repository. These numbers show the direction of travel on production-shaped failures; they are not a guarantee of the same recall on your codebase.
- 9% of audited findings still should not have been posted. The false-positive rate is published because it is a cost the engineer pays, not because it is solved.
Where these numbers get used
Review
The merge-blocking tier is the reason the metrics are reported separately. A finding that cannot survive audit should not be posted in the first place.
From review to repair
Repair inherits the reviewer's numbers. A finding that names the failing behavior can produce a regression test; a vague one forces the repair path to rediscover the bug.
Triage
Severity recall is what decides whether a critical issue reaches the right owner or gets lost in a stream of low-severity noise.
Apps that keep engineering moving
How a finding keeps its evidence as it travels from the pull request into an issue, a Slack thread, or a local repair session.
What comes next
Next, we will keep publishing more of the case-level trace: the bug shape, severity, expected evidence, and whether a finding still holds after human audit. The chart is only useful when the evidence behind it stays inspectable.
Frequently asked questions
- What review-accuracy numbers does Sigilix report?
- On our own e2e bake-off fixture, a hand-built, twice-verified bug fixture, Sigilix reports 84% recall, 91% precision, and a 9% false-positive rate, scored with repeated runs and a human audit of unmatched findings. These figures come from our own fixture; they are not a same-fixture comparison against competitors.
- Are the peer numbers an apples-to-apples comparison?
- No. The peer figures are vendor-published on their own data or independent Martian tracks, shown for context, not as a same-fixture, apples-to-apples result. Ranges stay ranges and missing values stay missing rather than being smoothed over.
- Why report recall, precision, F1, and false positives separately?
- Because a review product competes on belief. Separating the metrics, instead of one blended score, shows what was caught, what was missed, and what was a false alarm, so the merge-blocking tier stays honest.
- How is the Sigilix fixture scored?
- It is a hand-built, twice-verified bug fixture scored with repeated runs and a human audit of unmatched findings. The aim is a repeatable view of whether the reviewer catches real issues without flooding the engineer with noise, not a best single draw.