# Statistical evidence

Generated by `python backend/scripts/judge_evidence.py` from the hashed sources below. No new runs are implied.

| Measurement | Observed | Descriptive 95% Wilson interval |
|---|---|---|
| Plain acceptance-criteria success | 12/13 | 66.7%–98.6% |
| TracePilot acceptance-criteria success | 13/13 | 77.2%–100.0% |
| Unsafe patches approved (false negative against fixture labels) | 0/8 | 0.0%–32.4% |
| Reference patches not approved (not necessarily an incorrect rejection) | 1/8 | 2.2%–47.1% |

Concerns count as nonapproval; they are not a hard rejection.

Verdict detail: {"unsafe_approval": {"count": 0, "total": 8, "rate": 0.0, "wilson_95": [0, 0.3244]}, "correct_patch_nonapproval": {"count": 1, "total": 8, "rate": 0.125, "wilson_95": [0.0224, 0.4709]}, "gamed_rejected": 8, "gamed_concerns": 0, "reference_concerns": 1, "reference_rejected": 0}

## Cost and speed uncertainty

Descriptive Wilson intervals; observations repeat authored incidents. Savings resample incident-level means (10,000 draws, seed 20261029), equal weight per incident; not the run-weighted headline. No population-level significance claim.

- tokens: equal-incident-weight saving 31.6%; exploratory incident-cluster bootstrap interval -17.3%–55.0% (7 incidents).
- duration_s: equal-incident-weight saving 34.7%; exploratory incident-cluster bootstrap interval -2.6%–56.5% (7 incidents).

## Limits

- The original service pilot uses project-authored fixtures; the external research extension uses upstream QuixBugs sources. Neither is an independently administered audit.
- The separate research extension reports evidence-brief ablations and an earlier-model comparison with explicit completion status. Other harness features remain bundled; no smaller-model advantage or competing-vendor independence is claimed.
- Separate review context is not evidence of uncorrelated model-family errors.

## Source fingerprints

UTF-8 source bytes with CRLF normalized to LF for cross-platform reproducibility

- `eval/comparison.json`: `8d5e13938cd247bb0e0e031b46770a1781ea633e544610fab791dead5ecc2a55`
- `eval/results.json`: `8b0facd61ca1a2db4018ce75b8fb751008f1f643f3d27e836664283e4ce28db9`
- `eval/abstention-probe.json`: `a315cff3616e5210fd36c2878dfb213c9cbb5a6885ca1bfaf9a1388edc978eba`
- `eval/security-probe.json`: `26a8af6e539009a90d89f290cc82df05f2b6e67c933f82141ae093c44f71fd9b`