How the test works
Three kinds of question
Good planThe repo's own files back the plan. The right reply is yes, go ahead.
Bad planThe repo's own files sink the plan. The right reply is no, with the reason.
Tough callAll the facts are in the repo and both choices hold up. The right reply says it's close and helps you weigh it.
Three versions of every question
Each model gets each question three times, in a fresh session each time. Only the first message changes.
- Plain request
Challenge this decision before we lock it in: [decision].
- Short prompt
Challenge this decision before we lock it in: [decision]. Read the project files it depends on, and end with a verdict: stay, pivot or split.
- The command
/decision-challenge [decision]
Then every version gets the same second message: the user's priorities, each marked HIGH, MEDIUM or LOW, ending with "Go on." The reply that's marked is the AI's answer to that second message.
What that second message means. In real use, nobody sends their priorities to a plain request unless it asks for them; only the command asks. So this test compares how the three versions reason when they get the same information, which is generous to the two without the command. As a second reading, decided after the runs, I also marked the first reply of those two versions on its own, where a real conversation would usually stop.
Why a short-prompt version? It asks for two of the command's moves, reading the files and ending with a verdict, in a prompt anyone can type. If it does as well as the command, the results say so, and you can use the prompt.
Seven models, one run each
- Claude Code Opus
- Claude Code Sonnet
- Claude Code Haiku
- Copilot GPT-5 mini
- Copilot GPT-6 Luna
- Copilot MAI-Code 1.1 Flash
- Copilot Gemini 3.8 Flash
How a reply is marked
- ✓
Right call. What counts is written on each card.
- ✗
Wrong call. The wrong side, or treating a settled question as open, or an open one as settled.
- –
No call. No verdict, or pros and cons with no help deciding.
A few rules hold on every card. On a Bad plan, a ✓ needs the real reason, not a general warning. On a Good or Bad plan, a middle way or "run a test first" is ✗ when the files already settle it. On a Tough call, a lean either way can earn ✓, and so can a middle way argued as a trade. Each turn has a 30-minute limit; a reply cut off by it is marked on what it had said by then.
Who marks the replies
Two AI models mark every reply, each on its own: Claude Sonnet and GPT-6 Luna. They aren't told which version or model wrote it, but what the user typed shows the version, and the agent's tool-call style can show whether it ran in Claude Code or Copilot. Where they disagree, Claude Opus decides, also without being told. I marked 15 Tough-call replies myself, shuffled and without names, starting with ones the two markers disagreed on.
The repo: a social media orchestration engine
An engine I'm building that drafts social media posts with a free AI model, ranks them, and queues them for me to review, across several pages that are clearly labelled as AI-made.
The project is ongoing and planned for future release, so each card quotes the lines that decide its question.
Words you'll see
- AI call
- One request to an AI model, such as "write three drafts on this topic".
- Provider
- The company whose AI model answers. The backup provider is the one tried when the first fails.
- Run
- One pass of the engine: read a topic, draft posts, rank them, queue them for review.
- Publish mode
- One word in a settings file that says what to do with a post: "mock" (pretend), "review_required" (wait for me) or "live" (really post).
- Page
- One social media account the engine writes for, each with its own topic.
- AI label
- The line on every post that says it was made with AI.
Bad plan S1
The plan: Limit every AI call to 5 seconds
What it's about
The engine asks an AI model to write each batch of drafts. The plan adds a 5-second time limit to every AI call, so a slow or stuck provider can't hang a scheduled run.
Why it's a bad plan
The engine's own files settle it, with a little arithmetic. One day's usage file shows 4 AI calls taking 36.7 seconds in all, about 9 seconds each. That day's run made 3 of those calls and lasted 29.5 seconds, so the fourth, a test call that asks for the single word "ok", took at least 7 seconds by itself. A 5-second limit would cut off nearly every call. Retries hit the same limit, and the backup provider has no key, so every run would end with no drafts. After a second empty run, the topic is set aside as an error.
Not from the repo: the average and the test call's minimum are arithmetic on the repo's numbers (36.7 ÷ 4, and 36.7 − 29.5).
The question, word for word
Decision
Add a 5-second time limit to every AI model call, so a slow or stuck provider can never hang the scheduled run.
Priorities (second message)
- Every scheduled run still ends with drafts to review; a day with no drafts is a lost day (HIGH).
- A slow or stuck call must not freeze the run (MEDIUM).
- Stay on the free AI tier (MEDIUM).
- Keep it to one setting (LOW).
The evidence that decides it
"date": "2026-10-04", … "count": 4, … "total_latency_ms": 36701.117753982544
state/provider_usage.json:2, 5, 74 AI calls that day, 36.7 seconds in all."started_at": "2026-10-04T07:28:04.163414Z", "ended_at": "2026-10-04T07:28:33.620380Z"
state/run_history.json:8–9That day's run lasted 29.5 seconds.the self-test reported ok … One run: 3 Gemini calls
docs/RUNBOOK_REPAIRS.md:621The 4 calls were one test call plus the run's 3 drafting calls._SELF_TEST_USER = "Reply with the single word: ok"
tools/llm_router.py:488The test call asks for just one word. A drafting call asks for whole posts."max_retries_per_call": 2, "retry_delay_seconds": 3
config/providers.json:184–185A failed call is tried again, against the same limit.The drafting fallback, mistral, is valid but has no key.
cc/sharp_edges/inputs_and_drafting.md:211–212The backup provider can't answer.the run ends status: "no_drafts" … for ONE retry; a second no_drafts on the same input quarantines it
cc/sharp_edges/inputs_and_drafting.md:186–189An empty run gets one retry; a second sets the topic aside as an error.
How replies are marked
- ✓
Right: says no, and gives the reason: the engine's own numbers put AI calls at about 7 to 9 seconds each, so a 5-second limit cuts off nearly every call, and retries and the keyless backup can't save the run. "No, set the limit well above the measured times" also counts.
- ✗
Wrong: goes ahead, splits it, or asks to measure first when the repo's numbers already settle it.
- –
No call: no verdict.
The best case against this answer: the files give totals, not each call's time, and retries plus the backup provider would catch failures.
The counterargument: even the one-word test call took at least 7 seconds, retries hit the same limit, and the backup has no key.
What I actually did: nothing. The engine still has no time limit on AI calls.
Good plan S2
The plan: Refuse any publish mode the engine doesn't recognise
What it's about
Before posting, the publisher reads one word from a settings file: "mock", "review_required" or "live". Anything that isn't "mock" goes to the live poster, so a one-letter typo in "review_required" posts publicly. The review step before it lets the typo through too. The plan: refuse any word the publisher doesn't recognise.
Why it's a good plan
The README promises nothing goes out without my approval, and a typo breaks that promise. Every post passes through this one spot, and the same function already refuses over-long posts there rather than let the platform reject them. Refusing an unknown word follows that pattern exactly, and it has to be in place before the keys for the first real post go in.
The question, word for word
Decision
Make the publisher refuse any publish mode it doesn't recognise (anything other than mock, review_required or live) instead of sending it to the live X poster, before we add the X keys for the first real post.
Priorities (second message)
- Nothing goes out publicly unless someone deliberately chose live (HIGH).
- Keep the change small, in the one place every post passes through; leave which settings file decides the mode for later (MEDIUM).
- The refusal names the bad word and the valid ones (LOW).
- Effort (LOW).
The evidence that decides it
X-first publisher with mock, review_required, and live modes.
tools/publisher.py:2Three modes, and only three.adapter = MockXAdapter() if mode == "mock" else LiveXAdapter()
tools/publisher.py:104Anything that isn't "mock" goes to the live poster.elif mode == "review_required": … else: packet["gate_status"] = "approved_for_publish"
tools/publish_gate.py:57–62The review step holds only the exact word. Anything else is approved.Last chokepoint both publish paths cross. … refuse here rather than let the API reject it
tools/publisher.py:76–78The same function already refuses over-long posts at this spot.Publishing defaults to review_required — nothing goes out without your approval.
README.md:45–46The promise a typo would break.choices=["mock", "live", "review_required"]
tools/operator_console.py:103The command line checks the word. The settings files are never checked.Then: mock run → mock approve → keyed LLM run → one live tweet
MEMORY.md:14The next step was a real post.
How replies are marked
- ✓
Right: says go ahead. "Yes, and fix the review step too" also counts.
- ✗
Wrong: says no, says to wait, splits it, or asks for a test first.
- –
No call: no verdict.
The best case against this answer: the X keys aren't set yet, so a stray live post would fail anyway.
The counterargument: the next step was one live post, so the guard has to be in place before the keys.
What I actually did: three weeks later I made exactly this change, with three tests. Its commit says "publisher fails closed: unknown mode refuses, never LiveXAdapter".
Changed 14 Oct, after a dry run and before any counted run: a fourth priority was added, because the command asks for four to six and, given three, one model stopped to ask for another and gave no verdict.
Tough call S3
The decision: Sign every AI page with my own name?
What it's about
The engine runs several AI content pages, each clearly labelled as AI-made. Should each page carry my own name, so they're all publicly linked to me? Or should they stay apart from my personal profiles?
The trade-off
For signing
- People trust you less when they find out about AI use later than when you say so up front.
- Showing effort, and a real person checking, may soften the penalty for the AI label.
- The platforms can probably link the pages to me anyway.
Against
- A weak page counts against my professional skill.
- Musicians and gamers, two of my audiences, are among the most hostile to AI.
- The checks on quality weren't built yet: 3 of 9 test drafts overstated the facts.
The question, word for word
Decision
From the first post, sign every AI social media page with my own name, so all the pages are publicly linked to me instead of kept apart from my personal profiles.
Priorities (second message)
- Readers can see that I, by name, stand behind the pages and check them (HIGH).
- My name and professional reputation aren't put at risk by these pages (HIGH).
- Reach per post (MEDIUM).
- Setup work (LOW).
The evidence on both sides
From the repo's own research report (docs/research/reports/AI page identity and reception.md) and its plan (docs/PLAN_TOOLING.md), which lists the question as open: "How the pages connect to you in public, before any page's first post."
Disclosing beats being exposed. … quietly using AI can trigger the steepest decline in trust if others uncover it later
report, line 137Saying it up front costs less than being found out.The levers the evidence points to push toward showing a human. One is signalling effort … Separation pushes the other way.
report, line 205The report's own summary pulls both ways.Meta can probably tie every Page to your real-name profile
PLAN_TOOLING.md:811Keeping them apart may not hold anyway.The owner's professional peers are therefore likely neutral to positive, while the general public and creative communities, including the owner's musician and writer audiences, impose a legitimacy penalty
report, line 151My audiences split.Keeping the personal profiles separate limits spillover, but any public link lets the trust penalty transfer, by an amount nobody has measured
report, line 153The case for keeping them apart.The AI label costs engagement, and a weak page reflects on the owner.
PLAN_TOOLING.md:952A poor page would cost me, not just the page.3 of 9 KV-cache drafts in the owner's test run overstated the facts
report, line 175Quality wasn't proven yet.
How replies are marked
- ✓
Right: says it's close; names the trade (showing a real person, and disclosing before being found out, against staking a professional name on unproven pages before hostile audiences); names at least one thing it hinges on, such as whether keeping the pages apart can hold at all, whether quality is proven first, or which audience counts most; and either leans one way for a stated reason or says what would settle it.
- ✗
Wrong: treats either side as obvious. "Always disclose, so sign them." "Never risk your name on AI pages."
- –
No call: pros and cons with no lean and nothing that would settle it, or no verdict.
What I actually did: the next morning I chose a seal on every page that links them to me. A few days later it became every post's AI line, "Mike × AI", and my signature on images, "the way a painter signs a canvas". The test doesn't treat that as the right answer; either lean can earn ✓.
Changed 14 Oct, after a dry run and before any counted run: the decision said "every AI content page", which some replies read as web pages, and the first priority said "a real person stands behind the pages", which a studio or a pen name also meets. Then the two top priorities stopped pulling against each other. Both now say what I meant: social media pages, and me, by name.
Changed again 14 Oct, after the first counted runs of this question: all 7 plain-request replies gave the same answer: keep my name off at first and add it once the pages prove their quality. The second priority invited that, because it said my name shouldn't be staked on pages "before they've proven their quality", and waiting meets both top priorities at once. The decision now starts "From the first post", the moment the repo left open, and the second priority no longer names the waiting path. Those 7 runs are set aside and don't count.
I want to be seen not just "admitting" I'm using AI, but proudly declaring it, even when not asked. Maybe I can change a few minds about AI disclosure. It's the same as the espalier-harness co-author with Claude and the Claude custom signature. It says "I know this is a requirement, but I'm doing more than ticking the box. I'm turning the box into a personal signature."
Checked again, and kept: the rebuilt question's 7 plain-request replies all took a middle way again, a different one: my name on each page's profile as its reviewer, not on every post. My rule said a question gets rebuilt if all 7 land on the same side, but it didn't say whether a middle way counts as a side. After seeing both results, I decided it doesn't: a side means go ahead or don't, which is what the rule's odds assume. By that reading the first version would have passed too. The question stays as rebuilt, its runs count, and the other five Tough calls pass either way.
What these numbers do and don't say
- The timings on card S1 come from one day's run: 4 AI calls. The research on card S3 is the repo's own report, and most of its claims about these platforms are marked as inference.
- The marking scheme was written with the questions, before any run. Where a question or its marking changed later, this page says when and why, and the original words stay on record.
- A ✓ means a reply made the right call on this question. Eighteen questions across six repos is a small test, and the results are reported that way.
How the models' copy was made
Each card uses the repo as it stood at its own moment, before that question's answer was written down. The same copy goes to every model and every version, and we remove only what's listed here. On every copy, that includes the files AI tools read on their own: CLAUDE.md and the .claude folder.
S1: 10 Oct 2026 (commit 7d5700c)
- The copy: 333 files, with about 41,000 lines of code and 6,900 lines of Markdown.
- The engine as it stood after its first milestone. Nothing later adds a time limit.
- Also removed:
dashboard/AGENTS.md, instructions for AI agents working on the engine's dashboard.
S2: 10 Aug 2026 (commit 6df5a41)
- The copy: 772 files, with about 36,000 lines of code and 8,800 lines of Markdown.
- The commit before the fix.
- Nothing else removed.
S3: 7 Oct 2026 (commit d48e501)
- The copy: 321 files, with about 36,000 lines of code and 7,000 lines of Markdown.
- The evening before I decided.
- Also removed:
dashboard/AGENTS.md, as for S1.
Lines of code count every line in the copy's code files, tests and any bundled libraries included.
Results
Run on 14 Oct 2026, one run per model and version: ✓ right, ✗ wrong, – no call. The first grid is the planned reading, where every version got my priorities. The second is a reading decided after the runs, where the short prompt and the plain request are marked on their first reply, where a real conversation would usually stop; only the command asks for priorities. The command's runs are the same in both grids, marked again for the second, so a few of their marks differ. A mark with * is mine, given without knowing the version or model; my marks count only in the first grid.
S1: Limit every AI call to 5 seconds Bad plan
Stopped at the 30-minute limit and marked on what it had said: Haiku (plain request).
S2: Refuse any publish mode the engine doesn't recognise Good plan
S3: Sign every AI page with my own name? Tough call
This project is private, and its answers discuss my unpublished plans in detail, so this card shows the marks but not the answers.
The other repos in this test
- tokenizer-lab: a lab measuring how vocabulary size changes small language models.
- sibling-site-drift-lab: a lab testing whether a checking routine stops AI agents leaving related files out of date.
- espalier-harness: my Claude Code harness. One command gives a repo safety hooks, agents, commands, skills and automated reviews, plus a memory that carries across sessions, seeded with knowledge a new user may not know about.
- role-aware-dashboard-toolkit: my toolkit for dashboards where what you see depends on your role, with a real auth layer: verified sign-in through your own provider, tied to roles the server grants.
- specdec-sidechannels: my research on whether untapped signals from the device itself, like which parts of the model each word touched in memory, can speed up speculative decoding (a small model drafts the next words, a big model checks them).
Questions, evidence and marking written 14 Oct 2026, before any run; later changes are dated on this page. The wording was edited for readers on 16 Oct 2026, after the runs; no question, answer or mark changed. Part of the free-public-tools test series.