Agents that don't know they're in a factory: the graph moves out of the agents' heads

Everything I built before this post had one thing in common, and I only saw it once I had something to compare it with. In the mail-based factories the organization was free-form, and the agents knew it. They had roles. They knew the floor: who the lead was, whether the validator should write to the lead, what came after a push. The graph of activities that turns a need into delivered work lived in the agents' heads, and each agent's job included following it. When a thing went wrong, it went wrong in the graph: work duplicated, a verdict that never arrived, two agents thanking each other for six hours.

The new experiment is the opposite at the highest level, and that is the whole point of it. The graph is outside the Agents, and they don't know they're part of it. Each Agent is created to do one piece of agentic work for one Task, with a cleanly built Run Context, and it only edits files. Everything algorithmic is done by the Factory outside the Agents. The goal behind every design decision was to make decisions algorithmic, so that agentic work could not break the Factory's work graph.

I designed it with Claude Code, mostly by being interviewed. A grilling skill asked one question at a time while we kept a glossary and decision records, and the terms below are the glossary's. The repository is at github.com/yeiniel/agentic-software-factory. The earlier free-form work is on the free-form-agentic-factory branch, and this work is on experiment/parallel-implemen-write-acceptance.

The idea, and the tools that serve it

The idea has two halves. The first: an Agent should never be able to see, skip or misreport a step of the work graph, because the graph doesn't pass through it. In the old design every agent was asked to be honest about the flow. Here honesty about the flow isn't required of anybody. The second: when something fails, the Factory reacts to an algorithmic signal (an exit code, a verdict) and adjusts the next Agent's prompt, instead of hoping an Agent notices and fixes the process.

Several choices exist only to make the first half true:

• A fresh Environment per Agent. It is built from the target repository's own container definition and booted with a Run Context: the Task as a plain prompt, the role's system prompt and skills, and the project source as the workspace. The Agent starts clean, with nothing carried over to tell it where it is. An Environment never knows what runs in it.
• Dependencies installed in a build with network, then an agentic Environment that reaches only the model server. The Agent can't wander off into the world, and it can't turn its Environment into one the Integrator won't reproduce.
• The Factory provisions a branch for each role and records the Agent's result on it. The Agent never branches, commits or pushes, and decides nothing about where its work goes.

None of these is interesting alone. Together they make an Agent a function from a Run Context to a set of file changes.

The graph is a draft

To test the idea I needed some graph, so I hard-coded one in the Factory. That is for simplicity. A hard-coded graph is not the goal. The goal is a graph that is external to agentic behavior and as algorithmic as I can make it, one that keeps improving through feedback loops.

The current draft: a Task is a prompt describing a change to an existing repository, and it is done when the acceptance suite passes on main, which is the Deliverable. Each Run launches an Implementor and an Acceptance Writer in parallel, blind to each other. Acceptance Test Validation checks that the Acceptance Writer's new specs fail on the unchanged repository. The Integrator combines the two branches and runs the suite. Both of those are deterministic, with no network. If the suite is green the Factory publishes to main. Otherwise the failing Agent is retriggered with Feedback, or the Task ends Blocked, waiting for a human.

It already changed while I ran it, and every change was my decision, never something an Agent did:

• Acceptance Test Validation was split from the Integrator. My first Integrator also checked that the specs failed on the base. A Run showed why that was in the wrong place: the Acceptance Writer finished at 19:59, the Implementor hung until 20:16, and the Integrator never ran, so the Acceptance Writer's fault stayed invisible until the next Run. Checking its work needs only the base and its branch, so it became its own Activity, run right after the Acceptance Writer.
• Retries became nested loops. I first had only a cap of two Runs per Task. That is too coarse: an Agent that crashes and an Acceptance Writer whose specs are vacuous are different failures. Now each role has its own attempt limit within a Run, and the Run cap is the outer loop. After a failed integration only the Implementor is retriggered, with a fresh budget. Feedback says why in a fixed sentence, shows the raw output, and says the previous changes were discarded. The Agent restarts from the base code, with its session kept, and never sees the other Agent's branch.

What the idea survived, and where it leaked

I ran it on real open source issues. Each became a Task in the Task Store, the code lives on my own Repo Host, and nothing is sent upstream.

My first target was pydantic-extra-types, issue #326: punycode TLDs in a domain regex. The first runs failed, and the failures are the evidence for the idea.

• The Acceptance Writer changed the code under test. Its rule said never to. It did anyway, to check its own specs. Acceptance Test Validation caught it, because the specs passed on the unchanged repository, and the Acceptance Writer was told so in its Feedback. The rule had lived in a skill the Agent never opened: none of its 30 tool calls read it. I moved the everyday hard rules of a role into its system prompt and kept skills for conditional ones. But the point is that I didn't need the Agent to obey. The Factory was built to survive it not obeying.
• The Implementor couldn't write a backslash. The edit for a regex was damaged on the way out of the model, and the model then emitted closing tags until the timeout. Two Runs, both timed out, Task Blocked, main untouched. The Factory did what it should: it didn't integrate, and it said why.
• A broken Environment, my fault. The Agent reached for the system Python because the Environment's default interpreter lacked the dependencies. That was a bug in the Environment, and I fixed it there. I also split responsibility: the project's container definition says which tools it has and where, and the harness prompt says the Agent is offline.

With those fixed, the Task reached its Deliverable in one Run, in about 19 minutes. The Implementor made a one-line change. The Acceptance Writer added four punycode cases and, for the first time, edited only the tests.

The harder target was dcastil/tailwind-merge, a pnpm monorepo. Both Implementor attempts timed out after an hour and the Task ended Blocked, though combining the two branches by hand passed all 132 tests. The Agent had run pnpm test, which self-installed 560 packages and then chased failures in packages that the acceptance suite never checks.

That exposed a leak in the idea itself. The Agents had internet access. "You are offline" was a sentence in a prompt, and the network was a plain bridge. A property I wanted to hold by construction was held by wording. I wrote an ADR superseding the old one: the Agents' network is internal, and they reach the model server only through a small relay container. I checked it from inside the network: the model answers, the npm registry doesn't.

On unjs/ufo, issue #379, withTrailingSlash ignoring the path when the query ends with a slash, the Task reached its Deliverable in one Run on the isolated network, with 492 existing tests plus 7 new ones green. The loops did their job: the Acceptance Writer's first specs were rejected by Acceptance Test Validation, and its second attempt took 17 turns. The Implementor's first attempt had a working fix early and then spent the hour verifying it with throwaway tests, until the timeout. The second attempt finished in 22 turns. About 72 minutes, 60 of them that timeout.

That last one is a limit of the idea as I've built it. Nothing outside the Agent can tell it that it's done, so only a clock ends the over-verification. A signal from outside the Agent would be better than a line in its prompt.

The rest is simplicity

A few other choices matter, and are not what the experiment is about:

• The Task Store keeps Tasks as git-native issues in a repository.
• The Repo Host is a small git server with hooks, instead of a hosted service, so a push is the event that starts a Task.
• Pi with local inference for the Agents, on a 6 GB laptop GPU. I tuned the model server (-c 98304 -np 2, -ub 1024) and went from about 12.5 to about 18 tokens per second.

The Factory reaches the Repo Host, the Task Store and the Environment runner only through Adapters, each a swappable executable behind a command-line contract. That tells you these are implementation details, and not the core of the idea. I also test each product in the repository only at its seams, with full line and branch coverage, because I want to be able to rewrite the Factory and trust the suite. A read-only Console shows Tasks, their Runs and model speed.

What I take from this

• Decisions made by a program can't be talked out of. The Acceptance Writer broke its rule, the Implementor never converged, and the Factory still did the right thing, which was to not integrate.
• Each leak is a place where the Agent could see the graph. The skill the Acceptance Writer didn't read, the network the Agent shouldn't have had: both were places where I had trusted wording.
• Needs a signal for done. At the moment only failure and a timeout move the work along.

Where this is going

I still want a factory that learns from its mistakes. But learning looks different now. In the old design, learning meant an agent's memory and skills. Those still matter, but a factory whose Agents don't know they're in one has two other places where it can improve: the work graph and its feedback loops. Acceptance Test Validation appeared because a Run showed it was missing. A retry limit changed shape because two failures turned out to be different. Those are lessons too, and they live in the shape of the Factory and not in anyone's memory. A factory that learns is one that changes its graph too.

So the next question is whether a factory can change its own graph, and what evidence it should use when it does: which signals, how many Runs, and who signs the change off. For now I'm the one who does that. I'd like to find out how much of it can be turned into algorithm as well.

Next, I'll rerun tailwind-merge on the isolated network, and try a Task whose acceptance tests can't fail on the base, which this design doesn't yet support. Then, only then, my family's chore list.