Skip to content

Swap repair, for instances #21

Closed

Description

@sodre

Parent

#18

What to build

A swap is a CI runner taking a queued job other than the one it was launched for (see GLOSSARY.md and ADR-0003). Today a swap can strand a job:

  1. Runner R is launched for job Y.
  2. R takes job X.
  3. X's own message comes back from a deferral, and the launcher drops it as "not queued".
  4. Nothing ever launches a runner for Y.

A Spot reclaim after a swap is also reported on Y's commit, not X's.

To fix both, the webhook handles workflow_job in_progress. When the event's runner_name is one of the stack's runners and was minted for a different job, the webhook:

  • re-queues the job the runner was launched for, rebuilt from the runner's tags, so the launcher's "still queued?" check either launches it a runner or drops it;
  • re-tags the runner's instance with the job it really runs (repo, job id, run id, head sha, job name), so the notifier and the sweeper name the right job.

This ticket covers instances only. Tasks come with the Fargate tickets.

Acceptance criteria

  • in_progress on a runner running its own job does nothing.
  • in_progress on a stack runner that took another job re-queues the original job and re-tags the instance. Webhook tests cover both cases.
  • in_progress with a runner that isn't the stack's (no matching name or instance) is ignored.
  • The launcher's log, or the webhook's, records each repaired swap: which runner took which job, and which job was re-queued.
  • The webhook's role gains only ec2:DescribeInstances, plus ec2:CreateTags limited to instances tagged ci-runner:stack=<stack>.
  • Proven live: two ci-spot jobs queued together, with one deferred at the cap, both run.
  • docs/stack.md documents the in_progress handling, and the runbook's traps table gets the swap.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ready-for-agentFully specified, ready for an AFK agent

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions