Your agent can exit clean without writing a single line
Try this with whatever agent you use: give it a concrete task and, when it reports back that it's done, don't believe it. Go read the diff.
An agent can explore the repository, reason about the problem, write a convincing summary of what it would do, and exit with status zero without touching a file. It isn't lying: as far as it's concerned the run went fine. What lies is the criterion you're measuring it with.
That criterion — if the process returned no error, the task is done — is the one my orchestrator had by the second week of April, when the pipeline was already closing tasks in a chain. Written out like that it sounds reasonable. In practice it meant taking the suspect's word for it.
Four months and 2,095 tasks later it's still the hardest problem in the system, and I have no reason to think it's only mine.
The evidence is in the repository, not in the summary
The idea was simple and it changed the whole system: to mark something as done, you have to find evidence in the repository. Not read a claim in the agent's output. Find commits.
It sounds like a minor detail. It isn't, because it flips the burden of proof. Before, the system assumed work was done unless something failed. From that day on, it assumed work wasn't done until a real change showed up in the code.
Three days later I tightened the criterion further: the quality gate, until then a step you could skip, became mandatory between implementation and completion. It was no longer an optional check that ran when I remembered. Without a passing gate, the task didn't close.
The holes that showed up afterwards
What's interesting about this problem is that it wasn't solved by those two changes. It was solved — partially — over four months, and every hole I plugged taught me something about the difference between verifying and appearing to verify.
In early May I added a check comparing the files the spec declared it would touch against the ones that actually changed in the commits. Because finding "a commit" isn't enough: the agent might have committed something, somewhere, with nothing to do with what was asked.
That same day a case I hadn't anticipated appeared: an old gate verdict certifying new work. The task had passed the gate on an earlier attempt, was then modified, and the stored verdict still said everything was fine. The rule I added was to require a fresh gate before marking anything as done.
And there was a scoping problem, duller but just as important: the commit search only looked at the main repository. Since changes are spread across the backend and the various frontends, there were perfectly real implementations the system couldn't find, and tasks that got stuck for no reason. The search had to be extended to every repository in the workspace.
Four months on, the problem still surfaces
This is the part I least expected when I started.
On August 3rd — nearly four months after the first check — I found a new case: the gate was certifying tasks it had never actually looked at. It wasn't that verification failed; it was that on a certain code path verification never ran, and the result defaulted to passed.
That's the kind of bug that worries me most, because it's invisible. A check that fails makes noise. A check that doesn't run and returns green is indistinguishable from one that ran correctly, unless you go looking for it specifically.
Nine days later I took the most recent step: each task's implementation stage stopped being declared by what the system believes and started being derived from already-verified change evidence. In other words, state is no longer something written when a step finishes; it's something computed from what can be proven.
Looking at the project's full history, this is the only line of work running end to end across all four months. Everything else had a beginning and an end. This didn't.
Why it's the hardest problem in this space
I think there's an underlying reason, and it's that trusting is the default behaviour of almost any system orchestrating agents.
When you wire a model into a pipeline, what's naturally available to you is the process outcome: it finished, it didn't, it errored. Building real verification — going to the repository, finding commits, cross-referencing files against the spec, confirming the gate ran against this version and not another — is extra work nobody asks for and that goes unnoticed when it works.
It gets noticed when it's missing. And it gets noticed late, which is the worst part: a wrongly closed task doesn't produce an error, it produces a false foundation for everything built on top of it. The cost shows up weeks later, when something that depended on it doesn't work and nobody understands why.
So what I took from these four months isn't a technique but a criterion: trust in an agent isn't configured, it's built from evidence external to the agent. And useful evidence isn't that the process finished without error. It's that commits exist, with specific files, matching what was asked, verified after the last modification.
Everything else is taking the suspect's word for it.
What this made possible
Once the system could detect that a piece of work was wrong, the next question was exposed, and it was considerably more uncomfortable: now what do I do about it?
At that point, a failing verdict simply stopped the chain and waited for me. The agent had failed, the gate had caught it, and the system sat still until I showed up to look at what happened.
A pipeline that needs you watching isn't worth much. That's what the next entry is about: what the system does when an agent fails, and how to decide between pushing on, changing approach, or giving up.