Unattended Operation

Ballet is designed to run for long periods without a human present. That requires it to be safe to leave alone: it must not stall, loop, or burn money, and it must explain what happened.

Keep going

  • Independent work never waits. A question, failure or limit stops only the affected ticket and what depends on it.

  • Durable state. All pipeline state lives in the database; a restart of any Ballet service resumes work where it was (ADR-0016: Durable Orchestration in the Database).

  • Retries for infrastructure failures (an agent gone, network, provider errors) with backoff, distinct from agent-level failures.

  • Heartbeats. Agents report their own and each session’s liveness; missed heartbeats trigger reconciliation (APIs and Protocols).

  • Stuck detection. Sessions that exceed their time limit or stop producing output are terminated and retried, then escalated.

Stay within bounds

  • Budgets in tokens per ticket, per project per day and per organization per day (Budgets).

  • Concurrency limits globally, per organization and per project.

  • Iteration limits on pipeline loops.

  • Pause and kill switch per project and for the whole platform (Pause, resume and the kill switch).

Budgets

Budgets bound the tokens unattended work uses, as the LLM gateway reports them. Counted are input, output and cache-write tokens; cache reads are not (they cost a fraction). Set them on the organization page and in the project settings (0: no limit):

  • per ticket, over the ticket’s lifetime — the project’s limit, else the organization’s;

  • per day (UTC) for a project, and for an organization across its projects.

When work runs out of budget:

  • no new stage starts: a used-up ticket budget raises a blocking question (“raise the budget, then answer”); a used-up daily budget makes the flow wait (waiting: budget) until there is budget again — the next day or a raised limit — and the scheduler starts no tickets there;

  • the gateway refuses LLM calls of that scope (403 permission_error “Ballet budget exhausted: …”; it rechecks at most every 30 seconds), so a running session stops; its stage then waits for budget instead of failing.

Usage today is shown next to each budget.

Pause, resume and the kill switch

Humans can stop autonomous work at once — for a single ticket (move it to Paused), for a project (project page) or for everything (platform admins, on the home page):

Pause

No new stages start: the scheduler starts no tickets, queued runs are held, and a flow reaching its next stage — or its merge — waits (waiting: pause). Sessions already running finish their stage.

Stop all runs (kill switch)

Pauses like above and cancels every run in progress in the scope. A flow whose stage was cancelled this way runs that stage again in a new session after the resume.

Resume

Ends the pause: waiting flows continue, held runs start.

Pausing and resuming a project or the platform needs run.manage on that scope (organization admins for their projects, platform admins for everything). Every pause, kill and resume is recorded as an event (control.paused, control.killed, control.resumed).

Explain what happened

  • Every session has a timeline: stage, runtime, model, duration, tokens, cost, tools used, outcome, report.

  • A digest summarizes a period (e.g. overnight) per project: tickets completed, merged pull requests, open questions, failures, assumptions made, debt filed, spend.

See Observability.