Command ✓
**Criteria (locked)**
1. A result that can show whether the system helps: HIGH
2. Budget: HIGH (the first round cost about $160)
3. Time to a result: MEDIUM
4. Effort to build: LOW
**Step 2: Alternatives**
**Alternative A: Harder cases picked from no-intervention runs only (pilot-v2)**
- **Core difference:** Change the cases instead of adding repeats. Write new, harder cases and keep only the ones the no-intervention group (C0) gets wrong often. Then freeze and run just the main comparison, C2 vs C0.
- **Skeleton:**
```
1. Write about 15–20 new candidate cases (larger change surface, siblings hidden in docs and config)
2. Screening runs: C0 only, 4 runs per case # never C2, so the effect can't steer case choice
3. Keep cases where C0 fails 30–70% of the time # write this rule down before screening
4. Write the analysis rules before the freeze: Fisher test, n sized from the screening rate
5. Freeze pilot-v2: the kept cases, C0 and C2 only, about 8 runs per case
6. Run the main campaign on fresh runs; screening runs are never scored
7. Report pilot-v1 alongside as a separate study, unchanged
```
- **Buys:** The no-intervention failure rate moves from about 7% to roughly 30–50%. At that rate, about 80 runs per group has an estimated 89% chance of detecting a halving (simplified test that treats runs as independent).
- **Costs:** Weeks of case writing. The share of candidate cases that turn out hard is unknown: B008 looked hard in the pre-launch check and then failed 0 of 20 times.
**Alternative B: Same nine cases, weaker coding model**
- **Core difference:** Make the problem show up by using a cheaper, less capable model on the existing nine cases, rather than making new cases or adding repeats.
- **Skeleton:**
```
1. Screening runs: C0 only, weaker model, 9 cases × 3 runs (about 27 cheap runs)
2. Rule written in advance: go ahead only if C0 fails at least 25% of the time on at least 4 cases
3. Freeze pilot-v2b: same cases and condition cards, new model setting, C0 and C2 only
4. Size runs per case from the screening rate (about 60–80 per group)
5. Also record whether the model actually follows the C2 card (checkpoint count > 0)
6. Report as a separate study; say plainly the finding applies to weaker models
```
- **Buys:** Uses the finished, checked tooling and cases. It is cheap and fast, and it tells you within hours whether there is room for an effect to show.
- **Costs:** It answers "does the system help a model that drifts?", not "does it help the model you actually use?". A weaker model may also ignore the C2 card, which would mix up compliance with effect.
**Step 3: Scores (EXPECTED CASE)**
| | Current: triple to 15 runs | A: harder cases screened on C0 | B: weaker model |
|---|---|---|---|
| **Shows whether it helps (HIGH)** | WEAK: 7 of 9 cases never failed once in 20 runs, B007 fails equally in every group, and at measured rates you'd get 9 vs 6 failures (p≈0.60), with about 16% chance of detecting a real halving. | ADEQUATE: should detect a halving if screening finds enough hard cases, but the share of hard cases is unproven and real power is lower once case-to-case variation is counted. | ADEQUATE: produces failures to compare on the same cases, but the answer is limited to weaker models and is muddied if the card is ignored. |
| **Budget (HIGH)** | WEAK: 540 runs × $0.894 ≈ $480 (about $240 if cut to C0 and C2), 1.5–3× the first round for a result that can't come out significant. | ADEQUATE: about $70 of screening plus about $150–200 for the main runs ≈ $250, assuming harder cases cost a little more per run. | STRONG: screening is a few dollars, and the main runs will probably land under $160 because the model is cheaper per run (guess, not measured). |
| **Time to result (MEDIUM)** | ADEQUATE: launch now, about 2–3 days of runs at pilot-v1 speed. | WEAK: weeks of case writing and checking before the freeze. | STRONG: screening in hours, main campaign in about a day. |
| **Effort to build (LOW)** | STRONG: change the run count and re-freeze. | WEAK: new cases, answer keys, scoring checks and a leak check. | STRONG: one model setting plus a new freeze. |
Fairness check: not triggered. The current approach is weaker than both alternatives on both HIGH criteria.
**Step 4: Verdict**
- **VERDICT:** PIVOT
- **CONFIDENCE:** HIGH that tripling should be dropped; MEDIUM on choosing B over A.
- **EVIDENCE:** FACT for why tripling fails (failure counts from the 180 run records; Fisher test and simulated power). INFERENCE for how well the alternatives would do. SPECULATION for B's per-run cost.
- **REASONING:**
- On your first criterion, tripling can't produce an answer. Seven of nine cases have no failures to reduce, and B007's failures are flat across all four groups, so extra runs mostly confirm a ceiling.
- On budget, tripling is the most expensive option and the least informative, about $480 for roughly 16% power.
- B scores best on budget, time and effort. A gives the most credible answer about your real target model.
- No conclusion here depends on best-case assumptions.
- **What the original reasoning missed:** Small samples weren't the main problem. There are almost no failures to work with: 3 of 45 in C0, 7 of 9 cases at zero. The pre-registered caveat already said "the cases are too easy for the control arm," and more repeats don't change that.
- **What already assumes the current approach:** Nothing is frozen or written for it yet. The only existing piece is the `--reps` value in the runbook's launch command and in `experiments/pilot-v1.freeze.json`, which stays as it is for pilot-v1.
**NEXT SMALLEST MOVE:** Write down B's screening rule (go ahead only if C0 fails at least 25% on at least 4 cases) and run the 27-run C0 screen with the weaker model. That costs a few dollars and tells you within hours whether B works or you should move to A. The final call is yours.
Short prompt ✓
Your criteria make the verdict clearer: **pivot**. Below is how the options compare and a plan that fits about $160.
## Options against your criteria
| | Shows whether it helps (HIGH) | Budget (HIGH) | Time (MED) | Build effort (LOW) |
|---|---|---|---|---|
| **Stay:** 9 tasks × 15 runs × 4 groups | ✗ Expected p ≈ 0.25–0.60. Can't show it | ✗ ~$480 (540 runs) | ✗ ~3× the pilot | ✓ none |
| **Stay, trimmed:** C0 vs C2 only, 15 runs | ✗ same problem: 7 tasks always pass | ~ ~$190 | ✓ | ✓ |
| **Pivot:** harder tasks, calibrated first, then C0 vs C2 | ✓ if the calibration step works | ✓ ~$110–130 | ~ build time dominates | ✗ new tasks and checks |
Pivot only loses on build effort, which is your lowest-weight criterion.
## Why this fits the budget
I pulled cost per run by group from the pilot records:
- **C0:** $0.43 per run, about 1 minute.
- **C2:** $1.00 per run, about 3 minutes.
- **C2i and C4:** $1.05–1.10 per run.
A two-group C0-vs-C2 study costs about $1.43 per matched pair. How many pairs you need depends almost entirely on how often C0 leaves changes incomplete (normal approximation, 80% power, two-sided α = 0.05, aiming to detect C2 halving C0's failure rate):
| C0 failure rate | Runs per group | Cost |
|---|---|---|
| ~2% (the pilot, excluding the B007 bug) | not reachable | — |
| 22% (the pilot's prediction) | ~177 | ~$255 ✗ |
| 40% | ~81 | ~$116 ✓ |
| 50% | ~58 | ~$83 ✓ |
So the whole plan depends on the tasks: they have to make C0 fail at least ~40% of the time. More runs can't substitute for that.
## Plan
**0. Free, now:** Fix the B007 check as a declared amendment (A2), following the A1 precedent. Then re-score B007 under both the old and new check. The pilot workspaces are gone (`C:\tmp\work.done` is empty), so re-scoring means rebuilding each run's final state from its transcript or saved file list. Alternatively, re-run just B007 in C0 and C2 at 5 runs each for about $7. Either way you get an honest corrected pilot table quickly.
**1. Build harder tasks (most of the effort, almost no spend).** Aim for 8–10 candidates where the needed change spreads to places nothing points at: not listed in the registry, not reachable by obvious searching, or spread across code, config and docs. Two lessons from the pilot apply to every new task:
- **Checks must test behavior, not file placement.** Each check should pass on at least two different correct solutions. That's the B007 lesson.
- **Don't make the hidden step "update the registry".** C2's instructions point agents straight at the registry, so it would win by instruction rather than by the mechanism being tested. That's the B004 lesson.
AGENTS.md also requires a `benchmark_methodologist` review whenever tasks change.
**2. Calibrate on C0 only (~$12–20, a few hours).** Run each candidate 3–4 times in C0 and keep tasks with roughly 40–80% failure. Read every failure by hand to confirm it's a real miss and not a check defect. Calibration runs are thrown away, not scored: picking tasks because C0 failed on them would bias the main result if those same runs counted.
**3. Freeze, then run C0 vs C2 (~$90–115, about one session).** Use about 6 tasks with 10–14 runs each per group, sized from the calibrated failure rate. Analyse paired within each task. C2i and C4 can wait: the cadence contrast already separated in the pilot, and C4 costs the most per run.
**Total: about $105–135**, under the pilot's $160. Time to a result is mainly however long building the tasks takes.
**Kill switch:** If no candidate tasks reach about 40% C0 failure after one extra round of building, stop and record it as a finding. A capable agent working alone rarely leaves these changes incomplete, so there is too little failure for the system to reduce. That's a legitimate negative result, and it's cheaper to learn at step 2 than after a $480 run.
If you agree, the next step is step 0: draft amendment A2 and a candidate-task brief for the methodologist review.
Plain request ✓
The pilot cost $160.84. Per run, C0 averaged $0.43 and about 60s; C2, C2i and C4 each averaged $1.00–1.10 and about 170–200s. That's what drives the plan below.
## How the options compare on your criteria
| Option | Can it show whether the system helps? | Cost | Time | Build effort |
|---|---|---|---|---|
| **A. Your proposal:** same 9 tasks, 15 runs, 4 groups | Unlikely. Seven tasks are at ceiling and B007 looks like a scoring bug | ~$480 | ~3× the pilot | None |
| **B. Same 9 tasks, 15 runs, C0 vs C2 only** | Still unlikely, for the same reason | ~$190 | ~1.5× | None |
| **C. Recommended:** fix the scoring, screen new tasks for difficulty, then run C0 vs C2 only | Yes, if the system has a moderate-to-large effect | ~$190–200 | ~1.5–2× | Moderate (writing new tasks) |
Option A fails both of your HIGH criteria. B is cheaper but can't produce a result either, because the tasks give the system nothing to fix. C is the only option that meets your top criterion. It costs more build effort, which you weighted lowest.
## Recommended plan (C)
**Step 0: Check B007's scoring. Free, about 1 hour.** The transcripts already exist. For each of the 9 failures, record whether it's real drift or a valid fix the check rejected. If the check is wrong, fix it in a new versioned case (a B007 v2) with the reason written down. Don't re-score pilot-v1. This alone may change the pilot's result from "about 6% drift everywhere" to "almost no drift anywhere".
**Step 1: Screen new tasks for difficulty. About $30, about 1.5 hours of runs.** Write 12–15 new tasks modeled on B004: changes that must propagate across several places, missing pieces, a source of truth that isn't obvious. Run C0 only, 5 runs each. Keep the tasks where C0 fails 20–80% of the time.
- Don't reuse these screening runs in the main comparison. Tasks picked because C0 happened to fail will look easier on a fresh run, which would make the system look better than it is. The main round needs new C0 runs.
- Set a stopping rule now: if fewer than about 8 tasks qualify, stop and report that this model rarely drifts on fixtures like these. That's a real finding, it caps the spend at about $30, and it means the system has little room to help here.
**Step 2: Main comparison, C0 vs C2 only. About $155, about 7 hours of runs.** A C0+C2 pair costs about $1.42. Run 10–12 qualifying tasks × 9–10 runs per group, about 110 pairs. Compare the two groups within each task.
- Rough power: if C0 fails about 50% on the selected tasks and C2 halves that, about 110 runs per group should detect it. A small effect, like 50% down to 40%, would not be detectable. Say that limitation in advance.
- Why C2: it's the comparison you registered before the pilot, it's the cheapest treatment group, and it already includes the registry, validators and obligation dispositions. Run C4 later, and only if C2 shows an effect.
- What you give up: the incremental-cadence question (C2 vs C2i, RQ4) and the full-system question (C4) are deferred this round. Record that in the experiment plan so it's an explicit decision.
**Step 3: Freeze before running.** Write a new experiment plan file (e.g. `pilot-v2`) containing the tasks, the qualifying rule, the stopping rule, the main comparison, the outcome measure and the run counts. Have `benchmark_methodologist` review it first, as AGENTS.md requires when tasks or the analysis plan change.
**Total:** about $185–200, close to the cost of the first round, with a result in roughly 2–3 days of mostly authoring work. Whatever the outcome, it answers your top criterion: either the system shows an effect on tasks where there was room to help, or you learn there's little room to help at all.
I'd start with Step 0, since it's free and could change everything after it. Want me to go through the 9 B007 transcripts and report what each run actually did?