Twenty Rules

Every engineering organisation running coding agents has arrived at the same instinct. The agent did something wrong, so you write a rule. The rule file grows. It becomes the place where institutional knowledge goes, and eventually it becomes a document nobody reads and the agent no longer follows.

The instinct is wrong, and it is wrong in a measurable way. Instructions that do not interact survive in the hundreds. Instructions that constrain each other collapse at around twenty, taking adherence from roughly 96% down to 20%.¹ Real rule files are made of the second kind. Worse, when two rules conflict, nothing fails loudly. Anthropic’s own documentation says the model “may pick one arbitrarily”.² You get a silent, non-deterministic choice, and no error to tell you it happened.

The deeper problem is that rules were never an enforcement mechanism. The environment you build around an agent has two halves: guides that steer before it acts, and sensors that check after.³ Guides are advisory. Only sensors enforce. Most teams are pouring effort into the advisory half and calling it governance.

Takeaway: cap the behavioural guide somewhere between twenty and eighty rules and keep it under Anthropic’s published 200-line budget. When a rule keeps getting ignored, do not escalate it to capitals. Delete it from the guide and rewrite it as a check that runs. Then accept the limit of the whole apparatus: it regulates whether code is well-formed, not whether it is correct. Only a specification written independently of the code reaches correctness, and no amount of tooling substitutes for it.

---

1. The rule file is not a control system

Start with what is actually measured, because the field is thick with opinion and thin with numbers.

Three studies bracket the question of how many instructions an agent can carry, and at first glance they disagree by an order of magnitude. One benchmark scaled to 500 instructions and found the best frontier model still at 68% accuracy, with reasoning models holding near-perfect adherence to somewhere between 100 and 250.⁴ Another found that the rate of perfectly compliant responses hits zero by 80 instructions, across every model, format and placement it tested.⁵ A third found follow rates collapsing from about 96% to 20% across a range of just one to twenty.¹

The spread is not noise. It is the finding.

The permissive study used independent instructions, the kind where each rule can be satisfied without touching any other. The collapse study used interacting rules about format, tone and length, where satisfying one makes another harder. In that study a single innocuous-looking requirement, output JSON, turned out to be jointly unsatisfiable with nine other rules already in the set.

So the variable that matters is not how many rules you have. It is how densely they conflict. And the rules in a real CLAUDE.md or AGENTS.md are overwhelmingly the interacting kind: write tests this way, name things that way, never touch these files, always explain your reasoning, keep responses short. That puts the working ceiling for a behavioural guide closer to twenty or eighty than to two hundred and fifty.

Anthropic publishes the only concrete budget I can find from a credible source: “target under 200 lines per CLAUDE.md”.² The cross-vendor agents.md standard offers no length guidance at all.

2. Three ways a guide file rots

Size is only the first mechanism, and it is the least interesting one.

Dilution. Attention across a context window is not uniform, and reliability degrades as input grows.⁶ This is why a larger context window does not retire the problem. Window size is a claim about capacity, not about fidelity. One study of coding agents watched pass rates fall from 8 in 10 to 3 in 10 as input grew from 11,000 to 299,000 characters, and found that relevant context hurt just as much as irrelevant context.⁷ That last detail deserves a moment. The usual defence of a large rule file is that all of it matters. It does not help. Volume itself is the harm.

Conflict. Every rule you add carries a risk of being unsatisfiable alongside one already present. There is no compiler for this. Which rule wins varies wildly by model, with compliance under conflict ranging from 98.2% down to 20.5% across 37 models tested.⁸ And escalation does not rescue it: stronger warnings fix some categories of failure and leave others untouched “across all strictness levels”.⁸ Every time you promote a rule to ALWAYS or NEVER and it still gets ignored, you have paid permanent context cost for nothing.

Decay. Roughly 23% of 356 studied repositories already carry stale references to code elements in their documentation.⁹ A stale rule is worse than a missing one. It occupies context, it misdirects, and it can win the arbitrary coin-flip against a rule that is still correct.

There is a fourth problem, and it is the one that should worry a leader most: failure does not look like failure. In the coding-agent study, the agent maintained above 92% requirement coverage while producing artifacts that were invalid.⁷ A dashboard reporting 92% compliance would have looked healthy the entire time.

3. Guides advise. Sensors enforce.

The vocabulary here is worth adopting because it makes the decision obvious. Böckeler, writing for Thoughtworks in April 2026, splits everything around the model into guides, which anticipate behaviour and steer before the agent acts, and sensors, which observe after it acts and let it self-correct. Each is either computational, meaning deterministic and fast enough to run on every change, or inferential, meaning it runs an LLM and is slower, dearer and non-deterministic.³

Read that split as a cost and trust ladder and the budget allocates itself. Computational sensors are cheap enough to run constantly and reliable enough to believe. Inferential ones are neither, so you run them occasionally and treat their output as a suggestion.

Anthropic states the consequence for guides plainly: they are “not a hard enforcement layer”, and anything that genuinely must happen belongs in a hook.² The evidence agrees. In the same coding-agent study, an external checklist beat model self-checking by 10 out of 10 against 5 out of 10, and that was the only statistically significant result in the paper, more significant than its own headline finding about context length.⁷

This gives a clean operating rule. A rule that keeps getting ignored should be deleted from the guide and rewritten as a check that runs. Not emphasised. Not capitalised. Moved.

The same discipline applies on the sensor side, where over-constraint has its own price. Tightening a guardrail mostly buys you refusals rather than safety: benign refusals and genuinely warranted refusals correlate at 0.89, so the two move almost as one axis and few systems break the tradeoff.¹⁰

4. The limit nobody can engineer around

Here is the part that should temper any plan to reduce human review.

The harness regulates the shape of code, not its correctness. Computational sensors catch the structural material reliably: duplication, complexity, missing coverage, architectural drift, style. Neither they nor LLM-based sensors reliably catch the expensive failures, which Böckeler lists as misdiagnosis of the problem, overengineering, and misunderstood instructions. Her phrasing is careful and worth keeping: “They’ll sometimes catch them, but not reliably enough to reduce supervision.”³

Correctness sits behind a condition: it “is outside any sensor’s remit if the human didn’t clearly specify what they wanted in the first place”.³

That condition is the whole design problem. The obvious escape, having the agent write tests, fails for a structural reason. LLM-generated test oracles tend to encode what the code actually does rather than what it was supposed to do.¹¹ When the only input is the code, the test inherits the bug and certifies it. That is the default setup in most agent workflows.

The escape route exists but it is narrow. Recent work shows that oracles generated from an independent requirement specification track intended behaviour instead. The circularity is a property of the pipeline, not of the models. Something has to state the intent, and it cannot be the system being checked.

Nor can you delegate the judgment to another model. Nine frontier judges drawn from seven different model families supply only about two independent votes’ worth of information between them, because they make the same mistakes on the same items. The best single judge matches the whole panel.¹²

5. Where the money should go

Split harness investment by how long it lasts.

The durable half is about trust: permissions, sandboxing, deterministic hooks, evaluation suites, escalation design, context plumbing. This half has converged on a common architecture across independent SDKs, and it keeps growing rather than shrinking.

The disposable half is about capability: elaborate prompting, orchestration cleverness, workarounds for things the model cannot yet do. This half has roughly a ninety-day shelf life. The clearest warning on record comes from the team that invented one of the first serious agent scaffolds. They replaced it with about 100 lines of Python driving nothing but bash, and asked in their own documentation: “what if our agent was 100x simpler, and still worked nearly as well?”¹³ Anything you build to compensate for a model limitation, the next model absorbs.

The running costs are known and they are not the obstacle people expect. Anthropic reports around $13 per developer per active day and $150 to $250 per developer per month.¹⁴ Stripe merges over a thousand entirely agent-produced pull requests a week and caps CI at two rounds because runs “cost tokens, compute, and time”.¹⁵ Ramp reports roughly 30% of pull requests merged to its main repositories now come from its background agent.¹⁶ What nobody publishes is the cost of building and maintaining the harness itself, which is the figure a business case actually turns on.

Ownership, in every documented case, sits with a platform team rather than with each team separately.

6. What to do on Monday

1. Measure your guide file. Count the rules, not the lines, then count how many of them constrain each other. If interacting rules are past twenty, you are already in the collapse band.
2. Move the load-bearing rules into sensors. Anything you would be upset to see violated belongs in a hook, a linter, or a test, not in prose.
3. Delete stale rules on a schedule. They cost context and they win arguments against correct rules.
4. Stop escalating. If capitals were going to work, they would have worked already.
5. Write specifications independently of the code. This is the only lever that reaches correctness, and it is the one most teams have quietly stopped pulling.
6. Do not buy autonomy with any of it. A good harness lowers the floor on disasters. There is no evidence yet that it raises the ceiling on how much supervision you can safely remove.

The uncomfortable summary is that the harness is real engineering, worth staffing and worth funding, and it still does not answer the question you most want answered. It tells you the code is well-formed. It does not tell you the agent understood the job. Until intent is written down somewhere the agent did not author, that gap stays exactly where it is, and no number of rules will close it.

---

References

Footnotes

1. Anand & Chattaraj, arXiv 2608.02639. Follow rate falls from ~96% to ~20% across 1 to 20 stacked, mutually-constraining instructions; one “output JSON” rule proved jointly unsatisfiable with nine others. [primary, preprint] ↩ ↩²

2. Anthropic, Claude Code memory and settings documentation. “Target under 200 lines per CLAUDE.md”; “if two rules contradict, Claude may pick one arbitrarily”; guides are “not a hard enforcement layer”. [first-party lab docs] ↩ ↩² ↩³

3. Böckeler, B. (2 April 2026), Harness Engineering for Coding Agent Users, martinfowler.com. Guides (feedforward) versus sensors (feedback), cross-cut by computational versus inferential. Also the source of the shape-versus-correctness limit and the three failure modes sensors miss. Thoughtworks sells AI-delivery consulting; the article concedes its own central gap and reports no effectiveness numbers. [partial vendor flag] ↩ ↩² ↩³ ↩⁴

4. Jaroslawicz et al., IFScale, arXiv 2507.11538. 20 models against up to 500 instructions; best frontier model 68% accuracy at N=500; reasoning models hold near-perfect to 100-250. Instructions are independent keyword inclusions, which is why tolerance is high. [primary] ↩

5. Eliav, arXiv 2607.19257. Perfect-response rate reaches zero by N=80 across every model, format and placement tested; no advantage from markdown structuring. [primary, solo preprint] ↩

6. Hong, Troynikov & Huber (Chroma), Context Rot. 18 models; “performance grows increasingly unreliable as input length grows”. Chroma sells retrieval infrastructure. [vendor] ↩

7. Xue, arXiv 2607.17937. Pass rate 8/10 to 3/10 from 11k to 299k characters, with relevant context as harmful as irrelevant; external checklist 10/10 versus self-check 5/10 (p = 0.0325); above 92% requirement coverage maintained while producing invalid artifacts. n=10 per condition. [primary, preprint] ↩ ↩² ↩³

8. McCauley et al., IH-Benchmark, arXiv 2607.25987. 37 models; compliance under instruction conflict spans 98.2% to 20.5%; stronger warnings fix some failure classes while others “persist across all strictness levels”. [primary, preprint] ↩ ↩²

9. Treude & Baltes, arXiv 2606.09090. 23.0% of 356 repositories carry stale code-element references in documentation. Note this is a different phenomenon from context-length degradation despite both being called “context rot”. [primary, academic] ↩

10. Cui et al., OR-Bench, arXiv 2405.20947. Spearman ρ = 0.89 between benign-refusal and toxic-refusal rates. Cite the structural relationship rather than the per-model rates, which are from 2024-25 and now stale. [primary, academic] ↩

11. Konstantinou, Degiovanni & Papadakis, arXiv 2410.21136. LLM approaches are “prone on generating oracles that capture the actual program behaviour rather than the expected one”. Single model (GPT-3.5-Turbo), 24 Java repositories; the paper disclaims generality. 2026 follow-ups show specification-prompted oracles track intended behaviour, which is the basis for the independent-spec escape. [primary] ↩

12. Kohli, Nine Judges, Two Effective Votes, arXiv 2605.29800. Nine frontier judges across seven model families supply about two independent votes; roughly 75% of nominal independence lost to shared mistakes; best single judge matches the full panel. [primary] ↩

13. mini-swe-agent, SWE-agent/SWE-bench team (Princeton/Stanford). ~100 lines of Python, bash only, no tool-calling interface, above 74% on SWE-bench Verified. Note the honest comparison: the 2024 scaffold’s 12.5% figure was on SWE-bench full, not Verified, and most of the climb reflects model progress rather than scaffold minimalism. [primary] ↩

14. Anthropic, Claude Code cost documentation. “Around $13 per developer per active day and $150-250 per developer per month”, below $30 per active day for 90% of developers. [first-party, vendor] ↩

15. Gray, A. (9 February 2026), Stripe engineering. “Over a thousand pull requests merged each week” entirely agent-produced; CI capped at two rounds because runs “cost tokens, compute, and time”; selective testing because running three million tests per change is infeasible. [first-party, vendor] ↩

16. Ramp engineering (12 January 2026). “~30% of all pull requests merged to our frontend and backend repos” come from its background agent; built in-house for integration with the company’s own verification surfaces. [first-party, vendor] ↩