Repository navigation
Closed
Labels
ready-for-agentFully specified, ready for an AFK agentFully specified, ready for an AFK agent
Description
Parent
What to build
A swap is a CI runner taking a queued job other than the one it was launched for (see GLOSSARY.md and ADR-0003). Today a swap can strand a job:
- Runner R is launched for job Y.
- R takes job X.
- X's own message comes back from a deferral, and the launcher drops it as "not queued".
- Nothing ever launches a runner for Y.
A Spot reclaim after a swap is also reported on Y's commit, not X's.
To fix both, the webhook handles workflow_job in_progress. When the event's runner_name is one of the stack's runners and was minted for a different job, the webhook:
- re-queues the job the runner was launched for, rebuilt from the runner's tags, so the launcher's "still queued?" check either launches it a runner or drops it;
- re-tags the runner's instance with the job it really runs (repo, job id, run id, head sha, job name), so the notifier and the sweeper name the right job.
This ticket covers instances only. Tasks come with the Fargate tickets.
Acceptance criteria
-
in_progresson a runner running its own job does nothing. -
in_progresson a stack runner that took another job re-queues the original job and re-tags the instance. Webhook tests cover both cases. -
in_progresswith a runner that isn't the stack's (no matching name or instance) is ignored. - The launcher's log, or the webhook's, records each repaired swap: which runner took which job, and which job was re-queued.
- The webhook's role gains only
ec2:DescribeInstances, plusec2:CreateTagslimited to instances taggedci-runner:stack=<stack>. - Proven live: two
ci-spotjobs queued together, with one deferred at the cap, both run. -
docs/stack.mddocuments thein_progresshandling, and the runbook's traps table gets the swap.
Metadata
Metadata
Assignees
Labels
ready-for-agentFully specified, ready for an AFK agentFully specified, ready for an AFK agent
Activity
Merge ticket #21: swap repair for instances