Leaving the agent running all afternoon, not ten minutes
The difference between an agent that saves you time and one that eats it comes down to a single moment: what happens when the work comes out wrong.
If the answer is "it stops and waits for you", you have a tool that works only while you're watching it. You can leave it running for ten minutes, not for an afternoon. That's what I had by early May: the system could already detect that a piece of work was wrong, and what it did next was, essentially, nothing.
The problem wasn't technical. It was that I hadn't decided what should happen after a failing verdict. And there aren't two options there: refine what exists, pivot and start over, or surrender and leave a record. Having it written down when each one applies is what separates an experiment from something you can leave alone.
Separating the one who does from the one who judges
The first change was structural: explicitly separating two roles. One generates the work, the other evaluates it. This isn't an organisational nicety — it changes what information travels between steps.
As long as the evaluator returns "passed" or "failed", the only thing you can do with a failure is retry blindly. If instead it returns structured feedback, the system can decide what to do with it.
Evaluation became multidimensional. Instead of a single verdict, four axes: correctness, security, user experience and completeness, each with its own score. And something that turned out more useful than I expected: regression detection against previous attempts, to tell "still wrong" apart from "worse than before", which are different problems.
Three paths, not one
With that information in hand, the system picks between three exits. This is the part I think is missing from most orchestrators, where failure handling boils down to retrying.
- Refine: the work is on track but incomplete or has specific defects. It goes back to the same implementer with feedback about exactly what to fix.
- Pivot: the work is wrongly framed. Fixing on top of it won't help, because the starting point is the problem. The task is handed back a step up the chain.
- Surrender: after three attempts, the system stops pushing.
Surrendering isn't failing silently, which would be the worst of both worlds. It writes a structured incident report: what was attempted, with what feedback, which dimensions failed each time. That report is stored and becomes material to review later.
In fact, looking at the pile of those accumulated reports is how I came up with the most ambitious idea of the project, which is the subject of the next entry and turned out considerably worse than I expected.
The mistake that took me a month to see
There's a detail in this implementation that was wrong for a full month, and it illustrates how easy it is to waste context without noticing.
When the system decided to refine, it relaunched the implementer. But in doing so it discarded the previous session. The agent started from scratch: no memory of which files it had touched, which paths it had ruled out, or why it made the decisions it made. All it received was a block of text with the evaluator's feedback.
Put another way: I was asking it to fix work it didn't remember doing.
The fix, once I finally saw it, was one line: keep the session when the decision is to refine, and discard it only when pivoting, which is the one case where forgetting is deliberate — there you want the model starting without the bias of the failed attempt.
What makes that safe is a limit that already existed: since surrender cuts in at the third attempt, a refine with a live session can happen at most twice. Context doesn't grow indefinitely; it has a structural ceiling.
Telling apart failures that aren't the agent's
Something else I learned while operating this: not every failure is the worker's fault.
When the provider's API returned a network error, the system counted it as a failed attempt. The agent had done nothing wrong — it hadn't even started — but one of its three attempts was consumed. With two consecutive network blips, a perfectly healthy task reached surrender without anyone having evaluated anything.
Today those errors don't consume an attempt and are retried with backoff. Penalising the model for an infrastructure outage is, quite simply, measuring the wrong thing.
The fourth path
In August I added one more exit, for a case I hadn't foreseen: the impossible pivot.
Sometimes the evaluator determines the work is wrongly framed, but handing it a step up makes no sense because that previous step doesn't exist or is already exhausted. Before, that fell straight through to surrender. Now, instead of giving up, the system re-explores the terrain: it relaunches the code reconnaissance to rebuild context and retries from there.
It's an example of something that recurs throughout the project: failure policies aren't designed well up front. They're discovered by operating, when the case appears that doesn't fit any of the exits you'd anticipated.
What this buys you
A system with agents needs a failure policy as explicit as its happy path. And that policy has to distinguish three things that look alike at first: work that's close and deserves another pass, work that's wrongly framed and needs to go back, and work that has to be handed to a person.
Retrying without that distinction is burning money: you repeat the same mistake with the same information and expect a different outcome. And failing to separate surrender from the rest is worse still, because you lose the one case that actually deserved your attention.
With the three exits implemented, the system started resolving most of its own stumbles on its own. And what settled at the bottom were the reports of the ones it couldn't.
That pile of reports was the starting point for what comes next: asking the system to improve itself.