You used to run a 25-minute loop in a chat window: check on the queue, see if it stalled, fix it, kick it, repeat. That job now belongs to the machines. This page covers what the supervisor is, the three ways it failed to do that job this week, and the changes that landed today so the watching happens without you.
Your remaining role: read one Telegram card when something critical genuinely needs a decision.
The supervisor is a separate process, deliberately outside the moderator daemon, kept alive by launchd. Every 60 to 120 seconds it runs nine detectors against real state (the moderator's own database, process tables, health endpoints, git), writes an audit entry, takes recovery actions where policy allows, and pages Telegram only on new critical findings. A file lease guarantees exactly one supervisor runs at a time.
| Detector | Watches for | On finding |
|---|---|---|
| moderator-health | daemon down, crash-looping, hung, degraded-stuck | down/crashloop: restart immediately. Hung: restart after 3 confirmed ticks; live work defers it, but a 10-tick ceiling restarts anyway, capturing a CPU sample first |
| stalled-increment | a running increment with no live worker and stale commits | cancel the dead worker's plan; the controller re-pends it under its retry budget |
| wedged-epic | an active epic idle 30+ min with nothing committed | escalate (deliberate: with nothing running, a wrong automatic guess is worse than a page) |
| failed-merge | verified work stranded off the target branch | escalate |
| zombie-workers | orphaned worker processes | targeted SIGTERM of the proven-terminal pid |
| broken-toolchain | missing binaries (the node_modules pollution class) | escalate |
| mercury-health | Discord disconnects, auth failures, stale progress | freeze the one affected capability, never a full restart |
| self-health | its own audit or recovery hooks failing | escalate |
| livelock · NEW | the same increment dispatched 6+ times in 3h | stop the runaway epic + page you |
When a supervisor process got killed, its lease file still looked fresh. Every 10 seconds launchd spawned a replacement, which saw a "live" lease, logged duplicate_supervisor, and exited. Up to a full 10-minute TTL could pass with nobody on watch, and the cycle repeated for a day. Fix: a new instance now probes the recorded holder pid; a dead same-host holder is stolen from instantly. Gap shrinks from 10 minutes to 10 seconds, regardless of what killed the holder.
Every detector keyed on absence of activity: dead process, stale commits, idle epic. The 22-hour livelock had live workers, fresh commits, and completing plans, so it scored as healthy on every tick. Fix: the livelock detector reads the marathon's own dispatch ledger, where repetition-without-progress is unmistakable, and its recovery stops the runaway epic through the moderator's sanctioned endpoint. The daemon is healthy in a livelock; the epic is the patient.
When a daemon restart killed a just-started plan, the queue's policy was: mark the item failed, pause it, alert you, and stop the whole chain. That is precisely the "it keeps stopping and waits for me" experience. Fix: a failed plan now auto-retries in place, twice, salvage-first, before it is allowed to pause. Partial-honesty and no-progress verdicts still pause, because those deserve eyes.
Defense in depth, innermost first. Each layer assumes the ones inside it will sometimes fail.
| Layer | Watches | Acts by |
|---|---|---|
| marathon controller | its own verify gates and retries | bounded retry budgets on every failure path, then fail OPEN with a loud audit row, never a loop (99f8f72e) |
| queue runner | each dispatched plan's verdict | auto-retry failed plans twice, salvage-first, before pausing (c1b57f11) |
| supervisor | moderator, marathons, Mercury, workers, toolchain, itself, livelocks | capability freezes, confirmed-hung restarts, runaway-epic stops, Telegram pages (2d92d1f4) |
| deadman + launchd | the daemons and the supervisor as processes | respawn on exit, restart on failed health, KeepAlive |
| you | one Telegram card | only genuine decisions: money, releases, rejected designs |
The result observed live today: the moderator hung twice, the deadman bounced it, the queue's new retry re-dispatched the killed plan, and ME2 continued with no human touch. That exact sequence required three manual interventions yesterday.
The first version of this page listed three. Two were closed the same evening (commit cda4dfda): stalled increments and zombie workers now heal instead of paging, and hung restarts got a max-defer ceiling (the active-work guard counted pending queue rows as active, so a hung daemon on a box with a standing queue could defer forever: 321 detections, zero recoveries). Every hung-restart now also captures a 5-second CPU sample into data/diagnostics/ first, so the evidence stops dying with the restart. What genuinely remains:
tail -f tools/supervisor/supervisor.out.log in agent-orchestrator: one line per tick.tail -1 data/supervisor-audit.jsonl: the latest tick's findings and actions, machine-readable.cat data/supervisor.lease: who holds the watch right now (pid should be alive).