Execution Flight Recorder & replay
Observe every remote work-linked execution phase, enforce secret-free telemetry, cancel in-flight jobs safely, and replay failed runs.
Work execution admission invariants
When a work item triggers a remote connector or model execution via POST /api/v1/tenants/:tenantId/work/:workId/executions, D1 enforces strict admission invariants inside a single transaction before any message reaches Cloudflare Queues:
- Ready connector check: The target connector must belong to the tenant, be in
readystate, and target an exact allowlisted HTTPS origin. - Single active attempt per work item: At most one
queued,running, orretryingjob can exist per(tenant_id, work_id)pair. - Tenant concurrency cap: Active executions across a tenant cannot exceed its execution policy cap (default
4concurrent runs). - Transactional outbox: Job state and outbox payload commit atomically in D1 before Queue dispatch, preventing orphaned queue messages.
Ordered Flight Recorder timeline
Every admitted job emits ordered, trace-correlated events retrievable via GET /api/v1/tenants/:tenantId/jobs/:jobId/events:
{
"jobId": "job_01jcva89x2",
"traceId": "trc_7b31e9a0",
"events": [
{ "seq": 1, "phase": "admission", "status": "queued", "attempt": 0 },
{ "seq": 2, "phase": "outbox_dispatch", "status": "published", "attempt": 0 },
{ "seq": 3, "phase": "worker_claim", "status": "running", "attempt": 1 },
{ "seq": 4, "phase": "connector_call", "status": "succeeded", "attempt": 1, "httpStatus": 200, "durationMs": 412 },
{ "seq": 5, "phase": "completion", "status": "succeeded", "attempt": 1 }
]
} Zero-secret telemetry boundary
Flight Recorder events and structured Worker logs record tenant ID, job ID, trace/span IDs, attempt counts, HTTP status codes, latencies, and non-secret error codes. They never persist raw prompts, secret values, Authorization headers, queue payload bodies, or connector response bodies.
Audited cancellation and late-result discard
Operators can cancel an active execution at any time with POST /api/v1/tenants/:tenantId/jobs/:jobId/cancel:
- Queued jobs: Immediately marked
cancelled; queue consumers refuse to claim them when the message arrives. - Running jobs: Cancellation intent is persisted atomically. If an in-flight outbound connector request returns late, its result is discarded and the job transitions cleanly to terminal
cancelledstate with a Flight Recorder event.
Dead-letter recovery and immutable replay
Transient failures retry automatically on the same job with exponential backoff until the attempt ceiling is reached, at which point the job enters dead_letter.
To recover after fixing an upstream connector or quota issue, call POST /api/v1/tenants/:tenantId/jobs/:jobId/replay:
- Replay is permitted only for terminal
cancelledordead_letterwork-linked jobs—replaying asucceededjob is rejected with409. - The original failed job remains immutable in audit history; replay creates a new job ID linked to the source via
replay_of_job_id.
tests/backend/jobs.test.tstests/backend/audit.test.ts