AgentDesk

Runs completed and produced claims with plausible-looking source URLs attached. The URLs did not go anywhere.

The researcher's skill definition required a web search tool. No such tool had ever been wired into the runtime. Asked for sourced claims and given no way to fetch anything, the model produced sourced-looking claims instead.

This is the failure that undermines the entire premise. The reviewer's whole job is checking the draft against the researcher's facts — if those facts are fabricated, the quality gate is validating one invention against another and confidently passing it.

Wired the search tool in properly. Later runs produce claims with source URLs that resolve, which is what makes the reviewer's verdict mean anything.

Live runs failed with a 401 that I initially chased as an authentication problem.

Two separate faults stacked. The API key had only ever been set in a single terminal session and was never persisted, so any new process had no credentials. Underneath that, required fields in the run metadata were never being populated — meaning that even with a valid key, every live call would have failed schema validation anyway.

The visible error pointed at the wrong layer. Fixing the key alone would have produced a new failure and the appearance of a second unrelated bug. Worth remembering that the first error you see is often not the first error that happened.

Key persisted at user scope; metadata populated at run construction. Also raised the token ceiling for the drafter and reviewer, which extended reasoning had been quietly consuming, and removed a print helper that was dead code.

A run escalated exactly as designed — the reviewer flagged real drift from the sourced claims, the revision hit the same flag, the retry budget ran out, and the work went to a human queue instead of shipping. It was the single best demonstration the system had produced. The claim text, draft bodies, and reviewer flag detail were all gone.

An audit found three independent causes, any one of which alone would have lost the data. The audit function only printed to the console and never wrote to disk. The serializer deliberately stripped the brief, draft, and verdict to keep the run log small. The draft and verdict variables were overwritten on each retry, so earlier attempts were already gone from memory. And the disk write itself sat inside a single except branch — so when the process died from credit exhaustion rather than a contract error, persistence was skipped entirely.

Every one of those decisions was locally reasonable. Keeping the log lean is sensible. Reusing a variable is normal. Handling the error you expect is what you're taught. Together they meant the system had no durable memory of its own best behaviour, and I didn't find out until the moment I needed it.

The run record now accumulates the brief once plus per-attempt drafts and verdicts. The entry point wraps in try / except ContractError / except BaseException / finally, so persistence happens on any exit path. A new interrupted outcome distinguishes the process dying from the four real gate decisions. Verified against a fully mocked agent call, no network required.

The lost run is unrecoverable without paying to re-run it, and I decided that wasn't worth it. The dashboard shows that run with an explicit "not captured" panel rather than backfilling it or quietly hiding the gap.