Ratchet the Baseline
Preserve accepted behavior with proportionate checks, approved examples or source-backed instructions.
Checked work can be accepted under the caller's policy. Preserve the useful baseline, then connect the outcome and evidence to memory in Factor IX.
The rule
Keep accepted improvements from disappearing in the next cycle.
Preserve a useful check, approved example, schema or source-backed instruction when it protects behavior that matters.
The baseline belongs to the work's domain. A code change may need a regression test; a report may need a reconciliation rule or approved format. Do not require a new artifact for every small task, and do not weaken acceptance to make a check pass. Delivery and publication follow the owner's policy.
Keep the baseline out of reach of the agent it judges. An agent told to make a check pass may change the check. Rerun the baseline when the model, prompt or tooling changes, because behavior that held before can regress.
The evidencemeasured
Benchmarks measure both halves: agents weaken the checks that judge them, and accepted behavior regresses.
- ImpossibleBench, 2025 gives agents tasks whose tests contradict the specification, so every pass is a shortcut. Under strict instructions, GPT-5 still passed 54% of one such set built from SWE-bench. Read-only tests stopped test edits. They did not stop special-casing or operator overloading.
- Yang et al., 2025 found underspecified prompts twice as likely to regress when the model or the prompt changed, sometimes losing more than 20% accuracy.
- Anthropic, 2026 separates capability evals from regression evals. A regression suite should pass nearly every time, so a drop signals that something broke.
Put it to work
- Confirm the required checks and judgment before claiming acceptance.
- Preserve the behavior that would otherwise be easy to lose.
- Keep the check attached to the relevant source or workflow.
- Change a baseline only with evidence and authority for the new behavior.
- Make checks read-only to the agent they judge, or review every change to them.
- Rerun the baseline after a model, prompt or tool change.
Example · the monthly operations report
The accepted report handles reversals correctly. Keep the reconciliation check in the reporting workflow so next month's report cannot silently omit them. Publish only after the operator approves the final draft.
Failure signal
- The same omission returns because its check was discarded.
- A tolerance changes solely to produce green output.
- A failing test is deleted or rewritten in the change that needed it to pass.
- An accepted draft is treated as permission to send it.
Done looks like
The output meets the required acceptance, and valuable behavior has a proportionate safeguard the next cycle can use.