Building something that tells me the truth when I'm not watching

The starting point

I started FlowCrew with a specific request: I wanted to hand off a job that would run for hours, walk away, and return to a conclusion I could trust—including the conclusion that the job had failed.

The tools I had seen were capable of doing the work. The problem was that they were too eager to report success. I would come back to “done” and then discover that it was not done. For a task I was watching, that was an inconvenience. For a task I had deliberately stopped watching, it defeated the point.

That changed the question I was asking. I was no longer mainly interested in how to make agents do more. I wanted a system that could tell me the truth about what they had done.

Multiple roles were not enough

I began with the obvious architecture: a planner, coder, researcher, reviewer, and QA role, each responsible for a different kind of work. This is conventional. Most multi-agent frameworks have some version of the same arrangement.

But a chain of specialists has a blind spot. Everyone is working on a task; no one is watching the run. A stuck stage can consume its budget without attracting attention. Weak output can flow downstream because the next agent assumes its input is sound. A bad plan can keep being executed because the component able to revise it is not looking anymore.

So I added a supervisor whose object of attention is the run rather than any individual task. It samples progress at low cost and escalates to a real evaluation when something looks wrong. It can return inadequate work, end a stage that has been silent for too long, and force replanning when the direction is not converging.

The specialist asks, “What should I produce?” The supervisor asks, “Is this run still making credible progress?” Once the supervisor was watching the run, I still had to decide whether it could call the whole job finished.

Even the smartest component says done

The supervisor had the strongest judgment in the system, so the natural answer was to let it decide when the goal had been achieved. Then I measured what actually happened.

Across 2,989 runs with parseable records, 86 ended in reality_gate_failed. In 27 of those runs, the supervisor had already written a signal saying that the goal was achieved. Another 42 were stopped by checks the planner had written for its own goal: the crew had raised its own standard and then failed to meet it.

If the supervisor had been the final authority, those 27 runs would have been reported as successes. I do not read that result as evidence that one particular model was bad. The deeper problem is that the same population of models was writing the artifact, measuring the artifact, and judging the artifact. A natural-language declaration that the work had passed was not independent evidence. It was a fourth sample from the same distribution.

I therefore took final authority away from the model. A successful terminal result has to pass executable checks: scripts whose exit codes the engine can enforce. Only success is gated; failure does not have to be proved. Those checks can come from the human-written brief or from checks the planner writes for its own goal. The crew can raise the bar for itself, but nothing in the run can lower that bar.

The supervisor still matters. It can interpret progress, challenge work, and decide what deserves another attempt. It simply cannot turn its own confidence into proof of success.

The gate can move the goalposts too

An executable gate prevented the supervisor from declaring victory on its own, but it exposed another problem: the gate still wrote the adjudication. That meant the component reporting the result had several ways to make a disappointing measurement look acceptable without improving the work.

I made each adjudication answer to two things outside that report: the declared acceptance contract and the measurements recorded separately for that run. The engine mechanically rejects four paths:

1. Changing the metric name.
2. Quietly weakening the threshold or relaxing its direction.
3. Asserting a pass when the recorded number misses the requirement.
4. Returning an adjudication with no number at all.

These are structural rejections, not disagreements expressed in prose. The retry loop can act on them directly instead of waiting for a person to notice that the judgment no longer matches the contract.

The party writing the adjudication cannot also define what counts as passing. The model is free to explain the result, but the metric, direction, threshold, and recorded measurement constrain what that explanation is allowed to claim.

The counterintuitive result: constraints made it faster

Once several stages work in the same working tree, they can step on one another. My initial intuition was that stronger boundaries would buy safety at the cost of speed. A stage that had to stay inside a declared area, ask for anything else, and wait for a rule-based answer sounded more rigid than one that could simply work.

The measurements went the other way. I used the same engine; the only difference was whether the brief declared writable paths for each stage.

Did the brief declare stage scope?	Result
No	131 minutes; the gate’s first attempt was invalidated for writing out of scope, consuming 23.6 minutes
No	93 minutes; the gate’s second attempt violated scope and triggered replanning
Yes—the planner gave the gate 13 paths	37.6 minutes; it passed the gate in one round, and the repair stage was skipped entirely

Across 6 consecutive observations, there were 0 exceptions.

The result was not evidence that enforcement had become cheaper. Enforcement was already active in every case. The difference was whether the stage had been told, in an executable form, where it could work and how it could ask for more.

When the brief did not declare a scope, the planner gave the gate a scope of 0. Out-of-scope enforcement still rejected its writes. At the same time, the negotiation channel was not delivered because scope === undefined. The stage was caught between two facts it could not reconcile: it was not allowed to write, and it did not know that it could request permission. It kept colliding with a wall it could not see.

That failure mode changed how I thought about boundaries. A useful boundary has to be explicit, executable, and negotiable. Explicit means the stage receives a concrete scope rather than an implied convention. Executable means an out-of-scope write is not merely noted: the affected file is restored to the exact bytes it contained before that stage ran, then read again to verify that the restoration really happened. Negotiable means a stage that genuinely needs another file can ask for it.

The request is not decided by how persuasive the model sounds. It is answered by rules. One of those rules is that the requested file must not already have been modified. A stage cannot write first and reframe the violation as a request afterward. If a request is denied, the denial moves up into the next planning round. The stage does not keep hitting the same restriction until its budget is gone.

This is why I no longer see constraint and efficiency as opposites here. The slow runs were not slow because the boundary was strict. They were slow because authority had not been stated coherently: enforcement knew about the wall, while the stage did not know about the door. In the faster run, the boundary reduced ambiguity before work began. The 131-to-37.6 change is the practical consequence of making permission visible, enforceable, and open to a controlled form of negotiation.

Handoffs that can be checked

Every design above depends on the same property: there must be something concrete to inspect. If agents pass free-form text between them, the engine has to infer intent. That inference can be wrong, and there is no stable object to audit afterward.

I made each handoff a typed artifact, stored at a known place in the run directory, with an explicit producer and consumer. Plans, outputs, adjudications, scope requests, and approvals each have their own shape. If an artifact is malformed, the engine rejects it instead of guessing what its author probably meant.

One path still carried raw text: the path that redelivered a brief after a daemon restart. It was also the only path with this failure. A brief containing quotation marks was altered in transit. The restarted process computed a digest from bytes that differed from the bytes the operator had approved, so it refused to start. All 3 attempts failed. A byte-safe channel already existed in the code; it simply had never been connected to that path.

That path was the only one still carrying prose, and it was the only one that failed — which is the argument for the whole approach, made against my own code. Everywhere a handoff had a shape, a corruption like this would have been rejected at the boundary. Here there was no shape to violate, so the damage travelled until a digest finally caught it. The byte mismatch is what made the disagreement checkable at all: the process did not have to judge whether the altered brief was “close enough,” it could compare what it received against what had been approved and refuse.

I treat a handoff as a file, not a conversation. With files of known shapes, I can replay the whole lifecycle with scripted fake agents: zero models, zero tokens, and about one second. Whether the contracts are wired together correctly and whether the agents did good work become separate questions, and I can answer them separately.

The hardest judgment: when to stop

Open-ended research creates a different version of the same authority problem. Running an experiment is not the hardest part. The hardest part is admitting that there is nothing more worth extracting. Asking the agent currently running the experiments to make that call means asking it to judge whether its own work should continue.

I moved that decision into a pure function. A fixed policy computes keep or drop, then continue, ship, or ceiling, from the history of results. The policy, not the supervisor, owns the outcome. The supervisor can still evaluate what happened, but it cannot negotiate with the stopping rule.

The definition of improvement follows the same idea. A new result counts as progress only when it exceeds the uncertainty of that measurement. A favorable fluctuation cannot be booked as improvement. The reason for continuing or stopping is therefore checkable from the history rather than dependent on the confidence of the agent doing the search.

When this is not worth using

All of this machinery has a cost. For a single agent doing a bounded task, the gates, supervision, typed handoffs, scope enforcement, and stopping policy are pure overhead. I would use Codex or Claude Code directly.

Contracts also have to be written by a person. The change from 131 to 37.6 is double-edged: it shows the value of a clear scope, and it shows that the engine cannot rescue a bad brief. Mechanical enforcement can preserve a contract. It cannot supply the missing judgment that should have gone into the contract.

The case for FlowCrew begins only when the work will run longer than I am willing to watch it. At that point, a fluent completion message is not enough. When the same models produce, measure, and judge the work, “done” is another sample from the system that generated the work. I need something they cannot argue with: executable checks, fixed contracts, recorded measurements, explicit authority, and a policy that can say either ship or stop.

I never needed a system that always succeeds. I needed one I could leave alone and still trust when it reported failure.