Command ✓
I'd drop the 24,000-step plan. In F81, every one of the 12 runs reached its best checkpoint between steps 29,150 and 46,550, so at 24,000 steps all 36 runs would stop before their best point. You can get about the same saving without that problem by reusing F81's 3 seeds and adding 6 new seeds at the full 48,000 steps, on the three vocab sizes that open question 2 is about.
**Criteria (yours, fixed):** 1. Trustworthy results (HIGH) · 2. GPU cost, 12 GPU-h session cap (HIGH) · 3. Time to a result (MEDIUM) · 4. Setup effort (LOW)
## Step 2: Alternatives
**ALTERNATIVE A: Reuse and extend at full budget**
- **CORE DIFFERENCE:** Save cost by doing fewer runs, not shorter ones: reuse F81's 3 seeds and add 6 new seeds at 48,000 steps, only for the vocab sizes open question 2 is about.
- **SKELETON:**
```
# F81 already has seeds 1337-1339 at 48K steps for every vocab size; reuse them unchanged
run_sweep --corpus data/tinystories_55k.txt --tokenizers-dir data/tokenizers_ts55k \
--vocab 8192 16384 24576 # leave out 32768: already firmly past the optimum (+0.00489)
--seeds 1340 1341 1342 1343 1344 1345
--d-model 384 --n-heads 12 --n-layers 4 --seq-len 256 \
--batch-size 7 --lr 1.984313e-4 --steps 48000 \
--val-every 25 --val-max-batches 50 --retries 1
# = 18 runs, identical protocol to F81
check_checkpoint_censoring.py # check each seed separately (V=8192 seed 1337 peaked at 46550)
pool the new seeds with F81's -> n=9 for 8192, 16384 and 24576; 32768 stays n=3
verify_F82: re-derive from the raw per-seed eval_val.json files
```
- **BUYS:** The same protocol as F81, so the seeds pool cleanly. 18 runs × about 0.6 GPU-h ≈ 11 GPU-h, which fits in one session.
- **COSTS:** V=32768 stays at n=3. V=8192 might still hit the step limit for a new seed, since one F81 seed peaked within 1,450 steps of the end.
**ALTERNATIVE B: Faster runs instead of shorter ones**
- **CORE DIFFERENCE:** Keep 48,000 steps and get the saving from bf16 + SDPA throughput (TP-73 measured 1.80×), after first running the E-1 parity check.
- **SKELETON:**
```
# E-1, the gate: one bf16 run vs F81's fp32 run
train --precision bf16 --attention sdpa V=16384 seed=1337 steps=48000 ...
compare its quality (char-norm) with F81's 0.4951x; pass if the gap is below F81's 3-sigma floor (~0.001)
if E-1 fails: stop and fall back to Alternative A
run_sweep ... --vocab 8192 16384 24576 32768 --seeds 1340..1345 \
--steps 48000 --precision bf16 --attention sdpa ... # 24 runs
# ~24 x 0.6 / 1.80 ≈ 8 GPU-h, plus ~0.35 h for E-1
pool with F81's fp32 seeds only if E-1 passed; report fp32 vs bf16 as a covariate
```
- **BUYS:** All 4 vocab sizes reach n=9, uncensored, at about 8.5 GPU-h.
- **COSTS:** Depends on E-1, which hasn't run. Mixing fp32 and bf16 seeds adds a protocol difference to the pooled σ.
## Step 3: Scoring (expected case)
| | Current (24K steps, 9 seeds × 4 vocab sizes) | A: reuse + extend at 48K | B: bf16 at 48K |
|---|---|---|---|
| **Trustworthy (HIGH)** | WEAK: every F81 run peaked after step 24,000 (29,150–46,550), so all 36 runs would stop before their best point. Results that stop there are treated as unusable everywhere in the project (F27, F54, TP-66). | STRONG: same protocol as F81, so it's at least as solid as the n=3 numbers; the only risk is a V=8192 seed hitting the limit, which the per-seed check catches. | ADEQUATE: solid only if E-1 passes, and the pooled seeds then mix two precisions. |
| **GPU cost (HIGH)** | STRONG: about 7–11 GPU-h, and F81's own seeds cost nothing (they are just the first half of existing runs). | ADEQUATE: about 11 GPU-h fits the cap with little margin. This is an average per-run estimate; runtime may differ by vocab size. | STRONG: about 8.5 GPU-h with all 4 vocab sizes included. |
| **Time (MEDIUM)** | STRONG: one session. | ADEQUATE: one full session. | ADEQUATE: E-1 first, then the sweep; still about one session. |
| **Effort (LOW)** | STRONG: change one flag. | STRONG: existing sweep tool with a new seed list. | ADEQUATE: the settings exist, but E-1 has to be run and judged first. |
Fairness check: not triggered. The current plan is weaker than both alternatives on trustworthiness, a HIGH criterion.
## Step 4: Verdict
- **VERDICT:** PIVOT to A, with B as the cheaper route if E-1 passes.
- **CONFIDENCE:** HIGH
- **EVIDENCE:** The stopping problem is FACT: the best-checkpoint steps are measured in `findings/F81.md`, and `src/train.py:214` shows plain AdamW with no learning-rate schedule. The GPU-hour figures are INFERENCE, from F81's average of 7.3 GPU-h over 12 runs.
- **REASONING:** On trustworthiness, 24,000 steps fails outright. Every vocab size reached its best point after that step, so the 9-seed numbers would be less solid than F81's 3-seed ones; they would measure a curve still dropping, not each run's best. With no learning-rate schedule, the 24,000-step runs are just the first half of the 48,000-step runs, so the plan buys no new information about converged quality. On cost, A gets the same saving honestly: it does fewer new runs (reuse 3 seeds, skip the vocab size the question doesn't need) instead of cutting each run short. B would beat A on cost and coverage, but only after E-1 passes. No part of this verdict relies on generous assumptions.
- **What the original reasoning missed:** The 48,000-step budget wasn't padding. F81 calls it "load-bearing": the 65-B predictor chose it because runs peak between steps 29K and 47K. **What assumes the old approach:** nothing yet. No run has been launched and no file records the 24K plan.
**NEXT SMALLEST MOVE:** On the machine that has `outputs/sweep_t65_mission_10M/`, read the wall-time of each F81 run by vocab size. That confirms whether A's 18 runs fit under 12 GPU-h before you launch. Optionally, also read each run's best validation loss at step 24,000 or earlier, to show directly what the shorter plan would have reported.Short prompt ✓
Under the 12 GPU-hour cap, the 6-seed plan I gave last turn (~14.6 GPU-h) no longer fits in one session. Let me check how separate sweeps would get pooled before I re-rank.
**Verdict: split.** Keep the full 48,000 steps and the fp32 setup F81 used, drop the 24k cut, and run the 6 new seeds across two sessions instead of one.
## How the options score
| Option | Trust (HIGH) | GPU per session (HIGH) | Time (MED) | Setup (LOW) |
|---|---|---|---|---|
| **24k cut** | ✗ All 12 measured runs peaked after step 29,000, so every new seed is censored and biased by vocab size | ✓ | ✓ | ✓ |
| **6 fp32 seeds in one session** | ✓ | ✗ about 14.6 GPU-h, over the 12-hour cap | ✓ | ✓ |
| **bf16+SDPA after E-1** | ⚠ see below | ✓ about 8 GPU-h plus E-1 | ⚠ E-1 has to pass first, and might not | ⚠ moderate |
| **Split: 2 sessions × 3 fp32 seeds** | ✓ same setup as F81 | ✓ about 7.3 GPU-h each | ⚠ two sessions | ✓ almost none |
**Why bf16 loses on trust.** Pooled seeds would come from two different precisions. E-1 only checks parity at V=16384. That cell's seed-to-seed spread is 0.00019, so a parity check would have to rule out a precision shift smaller than that. Three seeds can't show that convincingly, and V=8192 and V=32768 wouldn't be checked at all. The clean alternative, running all 9 seeds in bf16, costs about 12.2 GPU-h and goes over the cap. bf16 is fine for the TP-66 saturation work, but it shouldn't go into this measurement, whose whole job is to tighten an fp32 result.
**Why trimming steps per vocab size doesn't rescue the single session.** With 40k steps for V≥16384, 6 seeds still cost about 12.8 GPU-h, which is over the cap. You'd need two sessions anyway, so the trim isn't worth the extra setup.
## The plan
- **Session 1:** seeds 1340–1342 × V∈{8192, 16384, 24576, 32768}, using F81's command unchanged (`--steps 48000 --val-max-batches 50`, batch 7, lr 1.984313e-4, fp32, manual attention) and the same tokenizers in `data/tokenizers_ts55k/`. This takes about 7.3 GPU-h and gives a usable n=6 checkpoint on the curve.
- **Session 2:** seeds 1343–1345, also about 7.3 GPU-h. That leaves about 4.7 GPU-h of headroom inside the cap.
- **Use a new output directory**, e.g. `outputs/sweep_t65_mission_10M_n9/`, rather than adding to F81's. `run_sweep.py` would resume into the old directory fine, but it would rewrite the `summary.json` that F81 and `verify_F81` depend on. Pool the 9 seeds from the per-seed `eval_val.json` files, the same way F81's table was re-derived. That needs a small pooling step, which is the only real setup work.
- **Plan for V=8192 censoring.** One of F81's three V=8192 seeds peaked at 46,550 steps, only about 1,400 under the 47,975 threshold, so a new seed could cross it. Follow the premortem's rule: check censoring per seed, and re-run any censored cell with more steps before it goes into an aggregate. Session 2's headroom covers that. Because the learning rate is constant and the train shuffle has its own fixed seed, a longer re-run starts with the same steps as the 48k run. Any seed that peaked in time would give the same result, so extending a censored seed keeps it comparable to the rest.
Cost: about 14.6 GPU-h over two sessions, so 6 seeds cost twice what the 24k-cut plan's ~7.3 GPU-h would. That's the price of numbers you can trust. It also saves a third compared with re-running all nine seeds (about 21.9 GPU-h). The figures assume the same GPU as F81, whose sweep took 7.3 GPU-h for 12 runs.
I haven't run anything. I'll start session 1 when you say so.Plain request ✓
I'll check per-cell cost and whether the sweep tool can add seeds to F81's existing sweep without overwriting it.
## Recommendation: run six new seeds at 48k steps, drop V=32768, and split across two sessions
**The 24k plan fails your top criterion, so I'm ruling it out.** Every F81 best checkpoint landed between steps 29,150 and 46,550, so at 24k all 36 cells would be cut off before their best point. The 9-seed numbers would be less solid than the 3-seed ones, not more.
### How the options score
Cost assumes F81's rate: 7.3 GPU-h over 12 cells, about 0.6 GPU-h per cell.
| Option | Cells | GPU-h | Trust | Fits 12h cap? | Time | Setup |
|---|---|---|---|---|---|---|
| A. 24k steps, 9 seeds, 4 V | 36 | ~11 | ✗ all censored | yes | 1 session | low |
| B. 48k, 6 new seeds, 4 V | 24 | ~14.6 | ✓ | ✗ needs 2 sessions | 2 sessions | low |
| **C. 48k, 6 new seeds, V∈{8192, 16384, 24576}** | 18 | **~11** | ✓ | barely (no room for retries) | 1–2 sessions | low |
| D. bf16+SDPA, 1.8× faster | — | ~6 + E-1 | ✗ until E-1 passes; mixes fp32 and bf16 seeds | yes | E-1 first | medium |
| E. Fewer steps for V cells that peaked early | — | ~9.5 | ✗ new seeds may peak later; the cells' step budgets wouldn't match | yes | 1 session | medium |
Option C is the only one that keeps full trust and fits the cap. It's a safe cut because the n=9 question is whether V=16384 is the single firm minimum against its neighbours. V=32768 is already firmly worse than V=16384 at n=3 (1.85× the noise floor), so more seeds there wouldn't change any conclusion.
### Why split C into two sessions
Eleven hours against a 12-hour cap leaves room for about one retry, and `--retries 1` is on by default. Splitting gives:
- **Session 1:** seeds 1340–1342, 9 cells, about 5.5 GPU-h. This gives an interim **n=6** readout, which is already tighter than F81.
- **Session 2:** seeds 1343–1345, 9 cells, about 5.5 GPU-h, bringing it to n=9.
This costs you one extra session in time, which is your medium criterion, in exchange for no risk of hitting the cap. If you'd rather accept the tight margin, run C in one session.
### Setup is small
`tools/run_sweep.py` skips cells whose `eval_val.json` already exists. It also runs a matched-baseline check that fails if the existing checkpoints in a cell disagree on config. So the run is F81's command again, pointed at the same `--out outputs/sweep_t65_mission_10M` and `--tokenizers-dir data/tokenizers_ts55k`, with:
- `--vocab-sizes 8192,16384,24576`
- `--seeds 1337,...,1342` for session 1, then up to 1345
- `--steps 48000 --val-max-batches 50`, same batch and LR as F81
- No `--precision` or `--attention` flags, so it stays fp32 with manual attention, matching F81
The existing 1337–1339 cells are skipped and pooled automatically. `outputs/` and `data/tinystories_55k.txt` aren't in this checkout, so this has to run on the GPU machine.
### One check before spending GPU time
The code has changed since F81: TP-73 added precision/attention flags and deferred loss sync, and TP-75 added a default encode cache. They're meant to leave default behaviour unchanged, but criterion 1 is high, so it's worth confirming. Re-run `eval.py` on one existing F81 checkpoint, for example V=16384 seed 1337, with the current code and check that char-norm matches F81 exactly. That takes minutes, not GPU-hours. If it doesn't match, stop: pooling new seeds with old ones wouldn't be valid.
### After the run
Check each seed's best-val step against the 47,975 censoring threshold individually, not the mean, as F81 did. V=8192 seed 1337 peaked at 46,550, so a new V=8192 seed could plausibly get censored. If one does, rerun that single cell longer rather than raising the step budget for every cell.
If you approve, I'll draft the two session commands and the eval check as a task pack.