Case study: every check was green and nothing was sending

This is the full account of an engagement summarised on the case study page. The client is a founder-led venture portfolio and is not named. One operator ran two businesses, one in DeFi growth and one in B2B services, on top of an open-source AI agent. We rebuilt that deployment into a production operations layer: high-volume automation with a human gate on everything irreversible, still run by a single person.

Where it started

The automation layer had grown by accretion on a fast-moving open-source agent framework. Nothing about it had been designed as a whole, and it showed in five places at once.

  • The agent burned millions of tokens a day polling channels that produced nothing.

  • The dashboards showed 46 of 46 checks green while zero emails were actually sending.

  • Six lifecycle email journeys had sat dark for a week.

  • The client’s fork had drifted 6,600 commits behind upstream.

  • At 18 to 30 notifications a day, the operator had stopped reading any of them.

Each of these is survivable on its own. Together they describe a system that is busy, expensive and unobserved: it costs money to run, it tells its owner that everything is fine, and the work it exists to do is not happening.

The shape every defect shared

Every real defect had the same shape: a component reported on itself, not on its output. A green scheduler says the job ran. It says nothing about whether the email sent, the post published or the money moved. Forty-six green checks were all true, and all beside the point.

That reframed the work. Health had to be redefined as a verified business outcome, and every irreversible action had to pass through a one-tap human gate. With those two things in place the machine could be trusted with volume while the operator kept judgement. Without them, more automation would only have produced more unobserved failure.

What we did

Audited every job against its output

We went through every scheduled job and compared what it claimed with what it produced. The always-on model pollers, the ones spending tokens to check channels where nothing had happened, were replaced with deterministic gates. A gate is ordinary code that checks whether there is real work, and the model wakes only when there is. The general pattern is set out in a separate transmission, Build token-optimised agent harness automations.

Replaced self-reported health with outcome contracts

Each job was given an outcome contract: a statement of the business result it exists to produce, with a check that fails when that result is missing, not when a process is. Canaries sit alongside the contracts. If the sends do not happen, the check goes red the same day, whatever the scheduler says about itself.

Made upstream adoption routine

A fork thousands of commits behind a fast-moving framework is a liability that grows every day. We built a twelve-test safety net that fails on stock upstream: it passes only when the behaviour the client depends on is present, so a rebase that loses any of it is caught before it lands. With the net in place, a thousand-commit rebase became a scripted, reversible operation. In total 6,619 upstream commits were rebased behind that gate, and the client now runs the update themselves.

One feed, one tap

The 18 to 30 daily pings were consolidated into a single actionable feed with a closed catalogue of one-tap approvals. Closed means the list of things the operator can be asked to approve is fixed and known, and nothing arrives that is not on it. Nothing publishes, sends or spends without the operator’s tap.

Instrumented revenue truth

Finally we instrumented the path from activity to money: person-level attribution from email to deposit, measured reply rates for each sequence, and a daily call sheet sized to the operator’s real capacity.

The figures

All six outcomes are verified.

  • Agent token burn from idle polling: eliminated, with about 3.7M tokens a day recovered.

  • Dormant lifecycle emails: re-lit, at 892 sends a day within 48 hours.

  • Operator notification load: from 18 to 30 a day down to under 3, all of them actionable.

  • Upstream drift: 6,619 commits rebased behind a test gate, with updates now routine.

  • False-green monitoring: eliminated. Health now means the output happened.

  • Qualified lead pool: four times the pool in one gated run, from 130 to 526 enriched leads.

What changed

The harness now runs both of the client’s businesses every day: content pipelines with validator gates, lifecycle email ladders under ramp caps, lead scoring on live hiring signals, and a morning call sheet that puts the operator on the phone with the twenty right people. Every send, post and spend still ends in a human tap. The machine builds the lists and the operator does the talking.

Health is measured at the output. If the sends do not happen, the client knows the same day, from one feed rather than thirty pings. The safety net and the update path were handed over as scripted, documented operations that the client runs without us.

What carries over

Three things apply to any deployment of agents, whatever the business. A process reporting on itself is not monitoring. Volume is safe only where the irreversible actions are gated. And a fork that cannot be updated will eventually be abandoned, so the update has to be made boring before it is needed.

Evidence

The client is a founder-led venture portfolio and is not named here. All six outcomes are verified. The short version, with the figures at a glance, is on the case study page.