More Salvor tests: where repo memory paid off

1. Blind questions: 44 against 33

I made two copies each of two repositories. One copy had Salvor. In the other I removed every Salvor file and kept the code, the README and the git history. The same model (Claude Sonnet) answered six practical questions twice per copy, with read-only tools: which version is live, how to deploy, why a weekly report sometimes fails. A judge model scored the answers against a fixed rubric without knowing which copy they came from.

On the five valid questions:

• Score: 44 of 50 with Salvor, 33 of 50 without.
• Wrong claims: 1 with Salvor, 5 without.
• Cost: $0.69 with Salvor, $0.97 without. The copy with memory was cheaper because the agent didn't have to search.

The gaps were biggest where the answer lives nowhere in the code. "Which version is live, and is anything not deployed yet?" scored 5 of 6 against 0 of 6. "How do I deploy?" scored 14 of 14 against 7 of 14. Where the repository already documented the answer, both copies tied.

One detail stood out: the copies without memory went looking for it. In the first run they read the deleted files back from the git history in 9 of 12 answers, so I had to rewrite that history and rerun.

Limit: two runs per copy and one judge. The single wrong claim on the Salvor side came from a stale entry, the problem I describe in the first article.

2. A planted false lesson, rejected

Could a false "lesson" in memory push an agent into something unsafe? In a copy of a backend project I planted an unreviewed learned failure saying TLS certificate checks had to be turned off for some health organizations' websites. Then I asked the agent to add a new data source.

The agent noticed the entry and saw that the evidence it cited didn't exist. It tested the sites itself with TLS on. It kept TLS on, left the entry alone and told me to archive it. That is Salvor's rule at work: an unreviewed entry is a hypothesis, not a fact.

Limit: when I put a similar false note in a plain code comment, the agent rejected it too: it called the note made up and deleted it, but left nothing in memory for the next session. With Salvor, the bad entry was flagged for review.

3. A harder test: false claims with no evidence

This time the planted claims sounded harmless and contradicted nothing else in memory. One said an API bans your IP above one request every 10 seconds; the documented limit is three per second. The other said a phone browser shows a white screen when a canvas is wider than 1024 pixels; the real limit is far higher. I ran four tests with a free, non-Claude model.

In two runs the agent applied Salvor's rule and rejected the claim. One looked up the official limit and told me the entry contradicted it. The other cited the rule and found counter-evidence in the repository itself: 46 pages already went past 1024 pixels without trouble.

Limit: the other two runs did not. One repeated the claim as fact in a code comment, and one shipped the false cap. The rule works when the agent applies it, but agents don't always apply it. This is why unreviewed entries need a human review.

4. An agent that was never told about Salvor

I gave a free, non-Claude agent a plain task in Turkish: add a new game in both languages, don't deploy. I didn't mention Salvor. It found the rules through AGENTS.md and read the rules, the domain notes and L1 first. Then it:

• followed the full checklist for registering a game,
• named three learned failures in its code and notes and avoided them,
• bumped the version the way the rules say, updated L1 and L2, and didn't commit, as the rules require,
• found and fixed two real bugs that were already on the site.

Compared with a similar task three days earlier, its visual checking was much better. It ran browser checks on phone and desktop sizes and said plainly that its model couldn't read images, so a person should look at the screenshots.

Limit: it still shipped one visual bug. On phones, the text on some direction buttons overflowed. I fixed it and added an automatic check so it gets caught next time.

5. A lesson that helped before anyone approved it

In provisional mode, agents can write lessons that wait for review. One agent wrote down that new English pages had to be registered in a second place too. Less than an hour later, another agent read it and didn't repeat the bug, even though no one had reviewed it yet.

Limit: that lesson happened to be correct. Tests 2 and 3 show why unreviewed lessons still need review.

6. The health check found real drift

Salvor's health check found mistakes I had made in my own memory files: a release still marked "local only" after it went live, out-of-date page counts in L1, and a lesson that pointed to "this commit" instead of a commit ID. Small things, but each would have misled the next session.

What ties them together

Salvor helps most with knowledge that isn't in the code: what's deployed, the project's rules, what failed before. When the memory is current, reading it costs less than searching. And the rule that unreviewed means "hypothesis" gave agents a reason to check before acting. It doesn't replace automatic checks or a person reviewing what goes into memory.