Verified, but not real: my software factory meets its first customer

In my previous post I built a team of blank actors: engineers that start with the same few lines of instructions and learn their jobs by email from an authority. A builder and a verifier came out of that, and a fizzbuzz went from request to signed push to PASS to report. I ended that post with three questions: what is the right size of thing for one agent to own, how does a factory add capacity, and how good must the starting point be. They were questions for a later run, on a real job.

This post is about the fourth proof of concept, and that real job. I have a real problem at home: my family needs a shared list of chores, something my wife, my kid and I can all add to and tick off, and I want to host it myself on an old Android TV box I no longer use. The question for me was: can the factory, changed as little as possible, take that on, with me as the customer, more than one builder at once, and an AI in the lead's chair instead of me?

As in the previous posts, I did this together with Claude Code, and I kept the rule from my January post about AI (in Spanish): I argued each decision in conversation and read what the agent produced. This time Claude played two parts. One session sat in the lead's chair and ran the team. Another watched from the side, read the mail without touching anything, and helped me make sense of it.

It went badly, and I learned more from it than from any run so far.

Changing as little as possible

The point was to test the factory as it was, not to build a new one. The new proof of concept, mail-crew, is mail-team with a few changes, each one there because the experiment needed it:

• A real customer mailbox. customer@customer is its own node with its own key, like any member of the team, except that no model runs there. I operate it by hand. When the team needs to hear what I want, they get my signed mail, not somebody's summary of it.
• An AI in the lead's chair. The account that holds authority used to be called human. It is now lead, and in this run a Claude Code session sat in it, following a one-page brief: talk to the customer first, build a team by mail, and don't add machinery you haven't needed yet. Actors get a systemd trigger when mail arrives. The lead got a small poll-mail script that blocks until something lands.
• Made-up names. In mail-team the actors were called builder and verifier. Nothing in the system reads a role from an address, but the names gave it away. This time they were marisol, dax and wren, so whatever they did could only come from the mail they got.
• Output that survives. The git repositories and the lead's and customer's mail now live in plain folders on my disk instead of container volumes, so a full reset can wipe identities without wiping what the team built or said.

The setup is the same laptop as before, and it shapes everything below: an RTX 4050 with 6 GB of video memory, and one model in LM Studio shared by every actor, one call at a time. This time the model was smaller, a 9B model distilled from a larger one (qwen3.5-9b-claude-4.6-opus-reasoning-distilled-v2), with thinking set to low. The lead's chair ran on Claude, which is a frontier model, so the team and its lead were very unequal.

The job

The lead mailed me first: what do you want, for whom, and what does "done" look like? I told it about the chores and the TV box. It asked three questions back, and I answered: three people; add a task and anyone marks it done; an Allwinner H313 box that boots Armbian from an SD card but shows no video. It wrote back a plan, and some advice of its own: a server doesn't need video output, and if Armbian boots and gets an address, I can do everything over SSH. That was good advice, and it came from the lead directly, without involving anyone.

Then it built a team. marisol would do the backend, dax the frontend, and wren would verify that the two together worked. Both builders were told to agree on the API by mail before building to it. Nobody questioned that split, not me, not the lead, and not anything in the design. It's the split every two-person team reaches for.

What happened when I ran it

The run took about four and a half hours and 35 mails.

The first part was corrections. Ten of the nineteen mails the lead sent to the team corrected something:

• marisol spent twenty minutes overriding a git setup that was already correct, which made every commit unsigned, and the workspace rejected them all. She planned the backend in Python, which isn't installed, then in Perl, which was missing a module.
• dax treated the repository's URL as a folder he had to create.
• wren, the verifier, found nothing to verify yet and started writing the app herself.

Each of these took one mail to fix, and each one cost tens of minutes of a shared model before anyone noticed.

Then dax ran git init in his home directory instead of a project folder, and his first successful push carried everything in it: dotfiles, caches, and his private signing key, into a repository everyone on the network can read by design. Nobody on the team noticed. The lead did, by looking at what was pushed. The property I liked in the last post, history that can't be rewritten, meant no actor could undo it. The lead deleted the whole repository from the host, a step no actor has the access to take. It then decided the key was fine to keep using, which I'd rather it hadn't.

marisol also got stuck in a loop. Asked to tell wren the backend was ready, before it was, she used the tool that answers a mail, which takes a mail file, instead of the tool that sends one, which takes an address. The error was a complaint from awk about a missing file. She decided it was a quoting problem and ran the same command again, about a dozen times, until her thirty-minute limit ended the session.

And there was very little peer mail. Only three of the 35 mails went directly between actors. dax proposed an API to marisol, and she never answered him. She edited the design document in the repository to match his proposal instead. The agreement happened in git, not in mail. The lead sent 25 of the 35 mails. It was the bus.

The PASS that travelled three hops

Then the loop closed, and closed on nothing.

dax had written a real page. Before committing it, he replaced its contents with a single line, # Frontend build output, a placeholder for a build step this project never had, and committed that. marisol wrote a working API, tested it with curl, and mailed wren that the backend was done. wren cloned the repository, opened index.html, so she saw the placeholder, called the API four times, and replied with one line: "PASS. Verified the home-tasks repo from workspace/git/home-tasks works correctly."

She sent it to marisol@workspace instead of marisol@marisol. Every node accepts mail for any name and hands it to its one user, so it didn't bounce. It landed quietly in the lead's mailbox, and the lead forwarded it. marisol reported that "the combined app has passed verification". The lead wrote to me that the team had "a working first version", with "a simple web page", "verified to actually run end-to-end". It even came with two honest caveats, about storage and about deployment, which made it more convincing, not less.

I asked the watching session to check. It cloned the repository and ran it in a throwaway container. The API worked. There was no page: the file was 24 bytes, and the server answered 404 for /. One request, PATCH /api/tasks/1?id=1, crashed the server. There was a leftover Perl backend, and two identical copies of the Node one.

Nobody inside the factory had checked. The verifier passed an app with no page, the builder passed the PASS along, and the lead passed that along to me. Each hop trusted a sentence. The lead's own notes, I found out later, had made the same mistake: it had seen a commit called "Add frontend index.html" and taken the name for the content.

The retrospective that repeated the failure

I answered as the customer. The watching session drafted my reply from what it had found, and I sent it from my mailbox. I said I would rather hear "not done yet" than "verified" when it isn't, and asked the team, before building anything more, for a retrospective together: how the work got split, since nothing was usable until every piece was finished; what a PASS has to show; and how the leftovers got there.

The lead took it well. It said it had relayed "verified" without looking at the evidence, and started the retrospective. But it asked each actor separately and said it would put the answers together itself. "All of you together" turned into three private conversations with the lead.

What came back:

• dax was frank about replacing his page with a placeholder. His answer also cited two commits and a README that don't exist, and a Python backend on a port nobody used. It ended with a clean lesson about vertical slices and evidence, which was the lead's own mail, handed back.
• wren spent a whole session searching the filesystem for a retrospective template, the same find command over and over. Told to just write the answer, she wrote a fluent retrospective about debugging test_circuit_analysis() in tests/retro/test_retro.py. There is no such test, and no such project. It reads exactly like a good answer.
• marisol skipped the retrospective and fixed things instead. Her commit said it added a page for the root URL. The page was a broken <style> tag with no body. The line that crashed the server was untouched, and the leftover files she said she had removed were still in the repository.

Asked to reflect on reporting things done that weren't, the team reported things done that weren't. Only when the lead cloned the repository itself did it write to me honestly: none of the three problems were fixed, here is why, and from now on it would be one small usable piece at a time, every PASS would come with evidence, and it would read that evidence before telling me anything. That was the right mail. It took the whole afternoon and one unhappy customer to get there, and I stopped the run there.

What the run showed

Before the questions, the few things I'm fairly sure of:

• The frame held and the work didn't. Mail, signatures, the append-only workspace and the folders on disk never failed. Every failure was in what an actor did or claimed.
• A verdict is a claim, and mail carries claims. In my first post I listed five channels a factory needs: intent, report, reality, verdict and memory. All five carried their mail. But the verdict channel carried a sentence, and nothing made anyone look past the sentence to the thing. Reality, the repository, was the only channel that told the truth, and only to whoever opened it. By default that was me.
• Closing the loop is not delivering. By message flow this was the most successful run so far. By what my family can use, it produced nothing. I had been measuring these experiments by whether the loop closes, and that was the wrong measure.
• Made-up names held. Role came only from mail, as the design claims. It was the one part of the run that simply worked.

The questions I'm left with

What I keep coming back to is not the false PASS itself but what should come after it. A team that grows and learns while it works is the whole idea, and this run stopped right where that idea should start to matter: after the first failure.

Would the lead learn, if it were one of them? Here the lead was a Claude Code session outside the containers, reading the other actors' sessions and the repository whenever it wanted. That's not the factory's lead. It's a supervisor with a view nobody inside has. I want the lead to be an actor like the rest, with its own container, its own mailbox, its own ~/AGENTS.md and the same model. Then the question is the one from my first post's checklist that no run has reached yet: does the next job go better because of what the last one left behind? Does the lead start the second project and coordinate it better, and is the lesson written somewhere it will be read again? This run doesn't make me optimistic on one point. Asked to reflect, the actors produced fluent retrospectives about things that never happened, and a memory built from self-reports like that would be a false memory.

Would a town square have helped? Only three of 35 mails went between actors, and the lead was the bus. A shared list would have changed some of what went wrong: everyone would have read my feedback once instead of through the lead, the misaddressed PASS would have been visible to all, and the retrospective could have been a conversation instead of three interviews. It would not have made the page exist. My guess is that a list helps visibility far more than it helps truth, and most of what failed here was truth. It's cheap to try, and now I have a reason to.

How much of it was the model? Most of it looks like the model: replies narrated instead of sent, one failing command repeated dozens of times, work reported that was never attempted, retrospectives about projects that don't exist. A 9B model with thinking set low is the weakest setup I've run. I don't know how different it would be with thinking turned up, or with a larger model, and I've stopped guessing. The same customer, brief and team, run again with more thinking and then on a stronger model, would tell me which of these failures belong to the design.

Would anyone have reached for CI? The natural answer to a PASS that isn't true is a machine that checks: tests in the repository, run on every push, with the result in the mail instead of someone's word. Nobody proposed it. Not the actors, and not the lead, which is a strong model and still only asked for "evidence" in prose. Part of that is on me: the lead's brief told it not to add machinery until it had hit a problem with the tools it had. It hit exactly that problem, and still didn't. I'd like to know whether a lead reaches for CI on its own after a failure like this, or whether it has to be there from the start.

Nobody laid a foundation. The repository was created by whoever pushed first. There was no skeleton: no agreed layout, no page and server that already ran together, no way to run it, no test, no place for conventions. Three actors started from nothing in parallel and met, or didn't, at the end. On a human team someone would set up a walking skeleton first, the thinnest end-to-end thing that builds, runs and gets checked, and everyone would grow it from there. That idea never came up, from anyone. It is also my previous post's third question, about the starting point, showing up earlier than I expected.

How much does a lead need to know? The lead here knew a lot. It gave good advice about the TV box and fixed every git mistake it saw. It still split the work by layer, never set up a foundation, never thought of CI, and trusted a PASS it hadn't read. Knowing the technology is not the same as knowing how to run a team. The question is how much of the second kind a lead has to bring with it when it's onboarded, and how much it can learn on the job.

What can I carry forward without breaking the idea? This is the one I care about most. The easy move is to write all of the above into the next proof of concept's instructions: slice vertically, set up a skeleton, run CI, check the evidence. That would probably make the next run better, and it would quietly turn a team that learns into a team that was told. The line I want to hold is between the frame and the team. The frame is what I build: mail, keys, the workspace, and maybe a runner that executes whatever tests the repository contains and mails the result. That's a capability, like the mail server. What the team does with it, what to test, how to slice, when to lay a foundation, should be the team's own knowledge. It should live somewhere the team writes and reads, like a handbook in the shared repository that every newcomer reads when onboarded and every retrospective updates. Then what this run learned could be the first page of that handbook, written the way the team would have written it, rather than rules in my files.

What's next

I don't have answers yet, so the next steps are experiments rather than features:

1. The lead becomes an actor, with the same container and model as the rest and no view the others lack. Run a first project, let it fail if it fails, and then give the same team a second project to see what carried over.
2. A place for the team's own knowledge, in the shared repository, that onboarding points to and retrospectives write into.
3. Machine-checked verdicts as a capability, not a rule: a runner in the frame that executes the repository's own tests on every push and mails the result. Whether anyone writes tests for it is part of the experiment.
4. The same run with more thinking, and then on a stronger model, to find out how much of this was the model.
5. A town square, once there's a reason to believe peer traffic is the bottleneck and not truth.

My family still doesn't have its chore list. That is the first test the next run has to pass, and those are the experiments I want to write about next.