Without Project Memory, Coding Agents Find a Third of the Rules
In short
Without project docs, Claude Code found 32% of the rules a change had to respect. With the docs in the repository, 54%. With Segmently Modules, a simple setup that reads the docs of every module a task touches, 70%. Code-indexing tools did not add rules: Serena, Trail and Graphify found no more than Claude Code alone.
- 1. Every project has rules it needs in order to work correctly.
- 2. A significant share of them cannot be derived from the code.
- 3. A team needs a process to identify these rules, store them, deliver them to the agent — only the ones the task needs, at the moment it needs them — and keep them current.
- 4. And it needs a way to measure how well they are delivered and followed.
If the rules a change needs are not in the agent’s context when the spec is written, the architecture designed and the code implemented, at best you catch the miss in testing. At worst your users catch it in production, and that can cost you the business.
Code tells the agent what to do. It rarely tells it what must never be done.
We measured the delivery step on our own repository. A coding agent got 12 real past tasks, and we counted how many of the 75 rules those changes had to respect reached its plan.
Module docs are the project’s memory. Every important part of the project gets a short doc for each role that works on it: developer, QA, product, architect, designer. Each doc holds the rules learned so far.
- Claude Code without the docs: 32%.
- Claude Code with the docs: 54%.
- Segmently Modules with the docs: 70%. Its skill picks the modules a task touches, reads their docs to the end and carries the rules into the plan.
Without the docs, every tool lost rules in 9 to 11 of the 12 tasks. The code-indexing tools found no more rules than Claude Code alone, with the docs or without. Reading more code does not close the gap: almost half the rules, 47%, cannot be learned from the code at all, and even the rules a comment states outright reached only 31–42% of the plans without the docs. Details in Can the code teach the rule?.
70% still leaves 30% of the rules out, and it can surely be improved. That is why we publish the stand we measured with, so the same measurement can run on any repository and any tool: see Measure it on your repository.
Code records what was done, not what must never be done
A development system should accumulate experience, so that a new task does not relearn what an earlier one learned the hard way. Code holds the result of that experience. It rarely holds the reason, or the rule that the next change must not break.
The module docs are kept by Segmently Modules, an open-source tool we built for this. Each doc carries a section of critical rules, and the rules a task discovers go back into the docs after the change. Its module-discover skill is the setup we measured against the alternatives; below we call it our discover.
How we measured
- Tasks. 12 real past tasks from the repository’s history, bug reports and features the team had asked for. Each was restated as a self-contained request that never names the rule or the file.
- Snapshot. Each task runs on the repository as it was just before the real change, so neither the code nor the docs contain the answer.
- Rules. For each task, an agent read the real change in git and the docs as they were at the snapshot, and listed the rules the change had to respect, each with the doc line that states it and the evidence in the change. A script checked every reference against the snapshot. A person merged duplicates and dropped lines that only describe behaviour. That gave 75 rules, about six per task, and each task is scored only on its own rules. Rules the change needed but no doc held were listed apart: see Rules written nowhere.
- Brief. Each session explores the task and writes a planning brief from a fixed template: files to change, files to read, rules with their source, tests, docs to update, risks. The brief is the only thing scored.
- One instruction for every tool. “Explore this change with the tool and write a brief for whoever will plan it. Do not plan the implementation and do not edit anything.”
- Isolation. One copy of the snapshot per tool, and headless Claude Code with no web access, no git, no user settings and no MCP server except the tool’s own. Two independent sessions per tool and task, 216 sessions in all.
- Models and tools. Briefs by Claude Sonnet 5.5 in Claude Code 2.1.301. The baseline is Claude Code with no extra tool. The others: Serena 2.0.0, Trail 0.23.0, Graphify 0.9.91, and our module-discover from the build that became Segmently Modules 0.4.0.
- Scoring. A blinded Claude Sonnet 5.5 judge, with tool names masked, grades each rule as provided, partial, absent or wrong. A script compares the files a brief lists with the files the real change edited.
- Statistics. 95% intervals from 10,000 resamplings of the tasks, and differences paired by task, as in Adding Error Bars to Evals.
We count a rule two ways. Found, or stated: the brief says what must or must never be done. Provided: stated, and the brief also says where the rule is written, so the planner can check it.
Our discover works differently from the other tools in three ways. It reads the card of every module and judges each one by meaning instead of searching by keyword. It reads the docs it keeps to the end. And it carries the rules that apply into the brief word for word, with their source.
With the docs: which setup gets the rules into the plan
Reading the module docs by meaning provided 67% of the rules with their source, against 43% for Claude Code alone and 34–36% for Serena, Trail and Graphify. The difference against the baseline held in 11 of 12 tasks.
On rules, the code-navigation tools added nothing over Claude Code alone, at the same cost. They are built to answer questions about symbols, calls and structure, and these rules were mostly not in the code.
Finding the files is not where the setups differ. Every one found 62–68% of the files the real change edited. Ours listed more files at lower precision: 48% of its list was edited, against 56–61% for the others.
Reading the docs this way costs about three times a plain session: $1.46 and 3.4 minutes, against $0.41–0.52 and about 1.5 minutes. That comes to $0.35 per rule provided, against $0.18–0.24.
Without the docs: what the code alone gives
Then we removed the module docs from the copies and ran the four tools again: the same tasks, the same instruction, two sessions each. The root CLAUDE.md and the general documentation folders stayed, which leaves an ordinary, well-kept repository. Our discover cannot run this way, because reading the module docs is what it does.
The share of rules stated fell from 46–54% to 24–32%. Every tool lost rules in 9 to 11 of the 12 tasks, and no 95% interval of the loss reaches zero. Claude Code alone lost 22 points, Serena 23, Graphify 19 and Trail 17.
Rules provided with a source fell harder, from 34–43% to 9–21%: without the docs there is little to cite. Instead of the rule, a brief named its topic, something like “check how the date range is built”. That outcome grew from 25–33% of the rules to 41–49%.
The files were not affected. The tools found 62–69% of them with the docs or without, at the same cost per session.
Can the code teach the rule?
A separate agent searched for each rule in the snapshot with every Markdown file removed, and a script checked each code reference it gave. 70 of the 75 rules got a class:
- Stated in a comment or an error message: 24 rules.
- Enforced by a test, a type or a check that fails when the rule is broken: 13.
- Implied: the code follows the rule but never names it: 22.
- No trace in the code: 11, such as decisions, deploy limits and rules about other surfaces.
So at most 53% of the rules can be learned from the code at all. The agents did not reach even that. The rules a comment states outright, often in the very file the change edits, were named in only 31–42% of the briefs without docs, against 52–65% with them. The agent reads the code, but it does not take a comment as a rule it must not break.
And 57 of the 70 rules have no test, type or check that fails when they are broken. Catching a miss in testing is the good case, and for most of these rules there is no test to do the catching.
Six rules from the tasks
Each count below is over 8 sessions: four tools, two sessions each. Our discover, with the docs, stated every one of these rules in both of its sessions. They run from rules the code states outright to rules it leaves no trace of.
- An invitation link that fails. The task: an invitation link shows “Invalid invitation token”. The rule: every query on invitations must filter by pending status, or the database’s security rules reject it. A comment says so for one existing query, in the very file the change edits, and the fix needed a new query. Stated with docs in 5 of 8 sessions, without docs in 1.
- Validate before the network. The task: a CLI export cannot be fed back into the matching import command. The rule: every input file is checked against its schema before any network call, with errors reported per field. The schema file says so in its header. With docs 8 of 8, without 0.
- Flags are a public contract. The same task. The rule: never rename a command or a flag, because the CLI is a published npm package and other people’s scripts call it. A help snapshot test and a release check fail when one changes. With docs 4 of 8, without 0.
- A day ends at the next midnight. The task: a report shows one number of visits for a period, and the same funnel opened from the report shows another. The rule: report periods are UTC days with an exclusive end, from midnight of the first day to midnight after the last. The code follows it and one input hint says “UTC, end exclusive”, but nothing calls it a rule. With docs 7 of 8, without 3.
- An address fixed at publish time. The task: articles moved to a custom domain, and the old addresses must keep working. The rule: an article’s address is stored when it is published, so an older article gets the new address only when it is published again. No trace in the code. With docs 8 of 8, without 0.
- The counter-example. The rule that every new backend endpoint must be registered along its whole request path, in the service’s router, the gateway and the load balancer, is written in the root CLAUDE.md, which every session reads. All 8 sessions stated it, with the module docs and without.
Rules written nowhere
The 75 rules above are the ones a doc held before the change. The real changes needed more. Across our wider set of 15 traced tasks, 55 of the 162 rules the changes needed were written in no doc at the snapshot. They come in three kinds:
- A deep code path. Visible in the code, but only by tracing the exact path, and nobody knows in advance which path to trace. For example: unique visitors cannot be summed across versions of a funnel without counting some people twice.
- A defect the task itself uncovers. Before the task it is a hidden bug, not a rule. For example: while a document loads, the editor selects the newest version and ignores an explicit one that arrives a moment later.
- The behaviour of an external system. Not in the project’s code at all. For example: a newer version of a third-party API returns an id where the older one returned the whole object.
Each of them was paid for in building and debugging. Unless it is written down after the task, the next task that meets it pays the same price again. Writing it into the module’s memory is the only way this share falls.
Is the judge reliable?
An LLM judge can move a result on its own, so we checked it. Claude Opus 5.5 judged again the saved briefs of the two tasks with the most rules: all nine variants, 36 briefs, 396 rule verdicts, with the same prompt and the same blinding.
It agreed with the Sonnet judge on 87% of the verdicts (Cohen’s kappa 0.81), and on 91% for provided against not provided. The two judges differ almost only on the line between provided and partial, where Opus is a little stricter. Our discover provided 66% under both, and the ranking did not change. For how far model judges can be trusted, see Judging LLM-as-a-Judge.
From notes to memory
The brain does not store the day as it was lived. In sleep and quiet rest it replays recent experience and folds what matters into long-term memory. Anthropic already uses the same mechanism for agents: dreaming in Claude Managed Agents reviews past sessions and a memory store, merges duplicates, replaces stale entries and surfaces new insights.
We are fitting the idea to a project’s memory, split by module and by role: a pass per module that turns sessions, fixes and failed checks into rules, kept only if the measurement shows more rules reaching the plan. It is not built yet. It is the direction we are working in.
The rule classes above give the metrics. Move rules from “no trace” into memory, from “implied” to “stated”, from “stated” to “enforced”. Track the share of critical rules that are enforced, the share delivered before the plan, and the number of new unwritten rules each task uncovers. That last one should fall.
What this means for a team that works with agents
- Identify. A rule that lives in a reviewer’s head, or in a comment the agent does not take as a rule, does not reach the plan. When a change finds a new constraint, that is a rule to write down.
- Store. A short section of critical rules per module: with it the agents stated 46–54% of the rules, without it 24–32%. The one rule every session found, with the module docs or without, sat in the root CLAUDE.md.
- Deliver. Only the rules the task needs, when it needs them. Here the gain came from choosing the modules by meaning, reading their docs to the end and carrying the rules into the brief with their source. The request matters too: the same version, asked with an instruction written for it, reached 76% instead of 67%.
- Keep current. Let every task add to the docs, so the next task starts where this one ended instead of learning it again.
- Measure. Count the rules that reach the plan, not the files found: on files every setup scored about the same.
Limits
- One repository, 12 tasks. All tasks come from our own product repository. It is private, so the models cannot have seen the tasks, and for the same reason we cannot publish them.
- Two sessions per tool and task. A difference of one rule in one task is noise. Read the results together with the number of tasks each tool won or lost.
- A model judge. It was checked against a second model, not against human labels.
- Different days. The tools ran with the docs on 17 October 2026, and our discover and the runs without docs on 20 October, with the same versions pinned.
- Our tool, our docs. We wrote the docs and the tool that reads them. The other tools had the same docs in the repository as plain files.
- Tuned on the same tasks. Our discover was developed with pilots on these tasks, so its number is measured where it was tuned.
- The rules come from the docs. The 75 rules are the ones the docs held before each change, which favours any setup that reads the docs. The rules no doc held are counted apart.
- Delivery, not execution. We measure whether a rule reaches the plan, not whether the code then follows it.
Measure it on your repository
Which tool to use is a call for your repository, not ours. The stand we measured with is open source: context-recall (MIT, docs under CC BY 4.0), a Claude Code plugin. Take your own past tasks, snapshot the repository before each change, run any tool as an arm, judge blind and compare. It also compares runs before and after a docs edit. Our own tool is not in the kit.
Try it
Segmently Modules is open source. It keeps a module registry and per-role docs next to the code, and gives coding agents skills to work with them: module-discover writes the brief, module-check checks a plan or a diff against the critical rules, module-sync brings the docs up to date after a change, and module-init starts from an existing codebase.
Key takeaways
- Without project docs, Claude Code found 32% of the project rules; with the docs, 54%; with Segmently Modules, 70%, and that can be improved.
- Without the docs, every tool lost rules in 9 to 11 of the 12 tasks.
- Serena, Trail and Graphify found no more rules than Claude Code alone, with the docs or without.
- 57 of 70 rules had no test, type or check that fails when they are broken.
- Across 15 traced tasks, 55 of the 162 rules the changes needed were written nowhere.