Dispatch Lifecycle
Claims, leases, the three-attempt budget, and what the reaper does and deliberately does not do.
A dispatched run becomes a queued task; runner machines poll to claim work they can serve; the claimer holds a refreshing lease while the container runs; the in-container daemon's authorized reports double as heartbeats. Launch failures requeue with backoff up to a budget of three attempts, then the run fails with the last error. A background reaper requeues claims that died before launching and fails launched runs whose lease is gone and whose daemon has been silent for ten minutes.
Any organization member can cancel a run that has not finished. An unclaimed queued run flips terminally to cancelled at once, before any agent starts; a claimed run carries a stop request that the runner honors at its next lease refresh. The stop request is durable: it survives launch-failure retries and lease loss, and the run always ends cancelled — never failed-with-retries, never requeued.
How to test it
| Suite | Proves |
|---|---|
pnpm vp run @firops/api#test — dispatch-queue.test.ts | Claims happen only when the image is ready and the provider is servable; cross-org claims are rejected; launch failures back off and exhaust the budget; lost leases requeue without corrupting state |
pnpm vp run @firops/api#test — runner-machines.test.ts | Claim tokens bind to machine and attempt; stale launch reports from a lost claim are rejected; lease errors carry machine-readable codes |
pnpm vp run @firops/api#test — kubernetes-listener.test.ts | The dashboard-managed runner drives the same claim/report loop over real HTTP, including that no org credential means no claim |
pnpm vp run @firops/api#test — runs-cancel.integration.test.ts | Member-level cancel over real HTTP: instant terminal flip for unclaimed work, cooperative stop for claimed work, no retry after cancel, 409 on double-cancel, and cancels surviving launch failures and dead runners |
Manual smoke: dispatch with devboxes listen stopped — the run stays Queued;
start it and watch the runner claim, launch, and report.
Decisions
- Task, run, and step move in one transaction, every transition a compare-and-swap — a crash between statements must never strand a terminal task under a still-running run, and a stale or replayed report must never overwrite a newer state.
- Reports and tokens bind to the attempt count — a delayed report from a previous claim of the same machine cannot clobber the current attempt's container identity.
- No reaper for queued tasks — a run nobody can serve waits visibly in the queue. Failing it would convert a capacity/credential gap into a false error; the queue state is the signal.
- A pending cancel outranks every requeue and failure path — once the user is told a cancellation was requested, no transient launch failure may put the task back in the queue (the next claim would erase the request) and no exhausted attempt budget may turn it into a failure. The stop resolves the run as cancelled, always.
- The reaper delivers a cancel no runner can — a launched run whose runner is permanently gone would otherwise stay alive on the container's daemon heartbeat forever. With the lease gone there is no renewal response left to carry the stop, so the server ends the run as cancelled itself; a returning runner removes the leftover container through startup reconciliation.
- A shared sweep watermark grants one full staleness window after an outage — every API replica observes the same recovery grace, so healthy daemons get time to report after a deploy instead of being falsely orphaned.
- The dashboard-managed runner speaks the exact same HTTP protocol as the open-source Docker runner — one machine registry, one claim endpoint, one report/lease/stop surface; there is no privileged internal path to keep in sync with the public one.
- Runner teardown is gated on structured error codes — only definitive answers (task gone, other organization, other machine) destroy a workload; a proxy hiccup or expired credential backs off instead of deleting a healthy container.
- The runner reports its CLI version on register and on every heartbeat — upgrade decisions need fleet-version visibility, and riding the calls the runner already makes means no separate telemetry path to run or secure.
- The API enforces the minimum CLI version with HTTP 426 and
listener_upgrade_required— the same code also rejects a claim when the runner and server share no launch protocol. Runners stop cleanly and leave active containers for startup reconciliation.
The credential side of claiming lives in Runners & provider credentials; the image gate lives in Runner images.
Next: Agent session events — what streams back out while the run executes.