Short memories: what my software factory's agents did when I made them forget between mails

In my last post I said the next thing to test was whether the team's lessons stuck. I'd also said, near the end, that I would try to re-inject memory after compaction. What I did instead was step back and ask what the two weeks of proofs of concept had taught me, and the first answer was uncomfortable.

On Oct 4 I asked Claude Code to list everything we had learned, reading the transcripts of both repositories. My own comment on the first list was: "either we learn a lot of shit or we didn't learn anything." I had never set up an experiment to learn about nspawn, for example, and several of the things on the list were bugs in my machinery, not findings about agents. So I fixed the frame before running anything else.

As before, I did this with Claude Code, arguing each decision and reading what the agents produced.

Fixing the frame first

• Fail loudly. A wrong address now fails when the mail is sent, with the reason, so the agent never believes a message went out when it didn't.
• Everything logged to one place. Each turn logs its start, its end and Pi's errors, and watch shows them.
• Pi waits for the model. A request queued behind another is slow, not broken, so the HTTP idle timeout is off.
• The model server unreachable from the nodes. After a reboot, the agents' containers could not reach llama-server by name. Recreating the pasta namespace fixed it. I recorded the cause and wrote a rule: the frame doesn't change mid-run.

The question: what is a session?

Until then I had run the team as one long conversation per agent, with all mail appended. I wrote down three questions I wanted to answer, in my own words: does better individual learning make a better factory, measured over a continued delivery process and not one run; if learning is really good, is one session per mail better; and what does collective learning look like for agents that aren't human?

So I built the memory machinery. Each agent got ~/.pi/agent/AGENTS.md as always-on memory, skills for situational procedures, and a consolidation turn after every batch of mail. I tested it first with isolated probes and then replayed the whole earlier scenario (one lead, one validator, two builders), with each batch of mail as a fresh conversation.

What the replay showed

• Context stayed small. Batches peaked at 3–12k tokens, with a median start near 3k. The first run climbed from 40–80k up to the ~82k compaction point. Compaction stopped being an issue by construction.
• Lessons did carry. One builder didn't repeat her own forgot-to-push mistake on the second feature. The lead's correction turned into a rule that applied elsewhere. The validator wrote a second skill distilling two hard lessons.
• The team was not faster. One small command, hub status, took roughly nine hours of in-world time. Part of that is four agents on two model slots. Part is correction overhead.
• The errors changed kind. I logged 14 distinct findings in the retrospective plus one small feature, against 21 for the first run's whole three-need arc. The first run's errors were collisions and false claims about shipped work. These were almost all about losing track of state between batches: a validator checking against a stale clone, then a clone fetched but never merged, then building tests from the code instead of the criteria he had read correctly; an agent signing another's name; a false "done" repeated by a second agent as a bigger false claim.

I stopped the replay at the plan stage of the third need. The result was clear enough, and my conclusion on Oct 6 was blunt: one mail, one session was the wrong bet.

What I take from this

• Memory doesn't replace a running conversation. It carried lessons, but not the thread of what had just happened. A long session keeps that for free.
• The validator needs its own discipline. Eli caught real bugs, including a format, sort and day-default bug in hub status that the builders' own checks missed. But three times his own checks were wrong in ways only I caught. Independent of the builder is necessary, not sufficient.
• A confounded run is half a run. Four agents on two slots, plus mistakes of mine, meant I couldn't cleanly separate the design from the noise. I flagged this in the findings.
• Rules kept only in wording get broken. This was already in my principles file. It comes back in the next post.

What's next

Switch the shape: one session per task, an agent working on several tasks at once, and learning as a separate process that reads sessions and doesn't run after every mail.

My family still doesn't have its chore list.