A review product has to compete on belief. It is not enough to post more comments or catch a bug once in a demo. The result has to survive audit: what was caught, what was missed, what was a false alarm, and which comparison is actually fair.
That is why Sigilix keeps benchmark evidence close to its source. The figure below separates recall, precision, F1, false positives, and severity recall instead of flattening them into one blended claim.
Benchmarks with the context kept intact.
Each metric opens into a native figure, with severity splits, published peer numbers, and ranges kept separate so the comparison stays easy to read.
The same comparison split by bug severity. Each mini-chart keeps the peer set together so the distribution is readable at a glance.
Critical
- Sigilix
- Greptile
- Bugbot
- Copilot
- CodeRabbit
- Graphite
High
- Sigilix
- Greptile
- Bugbot
- Copilot
- CodeRabbit
- Graphite
Medium + low
- Sigilix
- Greptile
- Bugbot
- Copilot
- CodeRabbit
- Graphite
How to read the fixture.
The Sigilix column is measured on a hand-built, twice-verified bug fixture, scored with K-repeated runs and a human audit of unmatched findings.
The goal is not a best draw. It is a repeatable view of whether the reviewer can catch real issues without flooding the engineer with noise.
Why the peer numbers stay bounded.
The peer columns are vendors' published figures or independent Martian tracks. They are useful context, but they are not presented as a same-fixture comparison.
Ranges stay as ranges. Missing values stay missing. That keeps the comparison readable without pretending the public data is cleaner than it is.
What the benchmark is for.
The numbers guide the product work: raise recall without weakening precision, reduce false positives by proof, and keep the merge-blocking tier honest.
Benchmarks are useful only if they make the next engineering decision clearer.
We built the benchmark backwards from trust.
A catch only counts when it preserves the engineer's reasoning trail: where the failure lives, why it matters, how severe it is, what the reviewer said, and which line-level evidence made the finding real. The set uses production-shaped failures: auth flows, audit logs, queue semantics, incident rules, metrics, and configuration edges.
Audit-log pagination skips records
The catch has to follow the cursor through filtering and show how a high-volume audit view can silently omit records a compliance review expects to see.
OAuth callback accepts missing state
The reviewer has to identify the unverified callback state and explain how the login flow can bind the wrong session or fail open.
Bulk-delete reports success on failure
The finding counts only if it follows the changed error contract to the UI or caller that now treats an incomplete destructive operation as successful.
Queue shutdown loses in-flight work
The review has to connect the exception handling path to jobs that are acknowledged before the worker can safely persist or retry them.
Incident rule reads stale config
The catch must trace the updated setting into the detector path and show where the old value still decides whether an incident is created.
Metric tags split the same shard
The reviewer has to point out that two tag names describe the same dimension, making dashboards and alerts undercount the affected shard.
Zero sampling rate is ignored
The catch has to show that an intentional 0.0 configuration is treated as missing, turning off the user's explicit sampling behavior.
What comes next
Next, we will keep publishing more of the case-level trace: the bug shape, severity, expected evidence, and whether a finding still holds after human audit. The chart is only useful when the evidence behind it stays inspectable.