# The follow-up tax: method

Written 22 Oct 2026, before any run. This file and `tasks.json` are
committed before the first run, so the git history shows the tasks were not
chosen after seeing results.

The findings of each run are summed up in `RESULTS.md`.

## The question

When you send an AI a rough prompt, how many follow-up messages does it
take until the answer is what you wanted? And how many does it take when
Perry rewrites the prompt first?

## The 20 tasks

`tasks.json`: 20 everyday requests (emails, a recipe, a workout plan, code,
a story, a toast). Each has:

- **draft**: what a person types, short and rough, the way people do.
- **needs**: three to five things the person actually wants from the
  answer but did not type (a length, a name, a deadline, a tone). Each
  need has a sentence the person types as a follow-up when the answer
  misses it.
- **facts**: what the person knows about their situation (names, dates,
  amounts). Used ONLY to fill the [blanks] Perry leaves in his prompt,
  the way a real person would type their own details into them. (Since
  the pilot, the needs count as known too; see "Changed after the pilot".)

Some needs Perry can't possibly know (that a dad already has every garden
tool). They are in on purpose: those cost a follow-up either way.

## Two ways of asking

- **As typed**: the draft goes to the chat AI unchanged.
- **With Perry**: the draft goes once through the live Perry function,
  exactly as the free extension sends it (no profile, chat app ChatGPT,
  history off). Where Perry left [blanks], a fill step types in what the
  person knows (their facts and what they want); a blank nothing fits
  stays as it is. The filled prompt goes to the chat AI.

Perry's profile ("he knows you") is OFF for every task. This measures the
rewrite alone.

## One conversation

1. The prompt goes to the chat AI (a plain chat, no system instructions).
2. A judge reads ONLY the answer and the list of needs, and marks each need
   met or not, quoting the answer as proof. It does not know which way the
   prompt was asked.
3. If every need is met, the conversation is done. Otherwise the person
   sends the follow-up sentence of the FIRST unmet need (in the order
   listed), and it goes back to step 2 with the new answer.
4. At most 5 follow-ups. A conversation still missing needs after 5 counts
   as "not reached".

One correction per message is a simplification; real people sometimes
list several at once. That is why the first answer is measured on its own
too (below), which doesn't depend on this rule.

## What is counted

- **Needs met by the first answer**, as a share of all needs.
- **Follow-ups** until every need is met (0 to 5, or not reached).
- **Words the person typed**: the draft, plus the facts typed into blanks
  (With Perry only), plus every follow-up. Filling blanks is work too, so
  it counts.

## Models and settings

- **Chat AI**: Google `gemini-3.5-flash`, default settings. It stands in
  for "the AI you chat with". It is NOT ChatGPT or Claude; results with
  those could differ, and the report says so.
- **Judge** and **fill step**: `gemini-3.5-flash`, temperature 0, strict
  rules (`judge.js`).
- **Perry**: the live function, the free plan's model, one call per task.
- Each task runs **3 times** per way of asking (the chat AI answers a
  little differently each time). Perry's prompt is the same in all three.
  Results are averaged.

## Rules we hold ourselves to

- The first full run after this commit is the result. Any later full run
  is published next to it, never instead of it.
- Every prompt, every answer, every judgement and every follow-up is
  published, including tasks where Perry did worse.
- Short pilot runs to test the harness are saved in `results/pilots/` and
  published too.
- Results describe THIS test only: these 20 tasks, this chat AI, this
  judge. Never quoted as "Perry saves X%" in general (§ 5 UWG). Any use on
  the website or in ads is cleared with the founder first and links to the
  full results.

## Changed after the pilot (22 Oct 2026)

The 2-task pilot (`results/pilots/`) found two harness problems. Both were
fixed before the full run, in the commit after the method one:

- The fill step lost a blank it should have filled ("[date]", when the
  person knows "since 3 November"): the model gave the blank back without
  its brackets. Blanks now go out numbered and come back by number.
- A real person filling "[number of days]" types the deadline they want,
  even though it is a wish, not a fact. So the person now knows their
  needs as well as their facts when filling blanks. Every word typed into
  a blank still counts as typed effort. A blank with nothing to fill
  stays in the prompt instead of being cut out, which had left broken
  sentences ("since , explain").

## Known limits

- We wrote the tasks, and we make Perry. The tasks are committed before
  the first run and published in full so anyone can judge them.
- The judge is a model and can be wrong. Its evidence quotes are in the
  report, so every mark can be checked by hand.
- The chat AI and the judge are the same model family as Perry
  (Google Gemini).
- 20 tasks is a small sample.
- The fill step is a model playing the person. Every value it typed is in
  the report.

## Running it

```
node tools/followup-tax/run.mjs            the full run (60 conversations per way)
node tools/followup-tax/run.mjs --pilot 2  a pilot on the first 2 tasks, 1 repeat
```

Needs `GEMINI_API_KEY` in the local `.env` (git-ignored). Every Perry call
uses a throwaway `eval-` device id and counts toward the per-network daily
cap, so one full run a day at most.