Skip to content

Watch for the first real Fargate Spot reclaim and sweeper stop #43

Open

Description

@sodre

Parent

#18 (follow-up of #31)

What to build

Nothing to build. Watch for the first real occurrence of the two #18 behaviours that can't be forced, and record it here.

  1. A real Fargate Spot reclaim. The notifier logs reported <task id>: <repo> job <id>, and a failed check run ci-runner / <job name> ("Spot capacity reclaimed: re-run this job") appears on the commit of the job that was actually lost, under its real job name. Only synthetic SpotInterruption events against real tasks have been tested so far (Sweeper, notifier and swap repair for tasks #30). An ignored: <task id> is no job task line means the tags couldn't be read.
  2. A real sweeper stop. The sweeper stops a Fargate task launched more than 2 hours ago, logs it among stopped, and on a later run removes that task's offline registration. Only clean sweeper runs over younger tasks have been seen (Sweeper, notifier and swap repair for tasks #30), and StopTask under the role's cluster condition is only simulated.

Where to look: aws logs filter-log-events --log-group-name /aws/lambda/ci-runner-notifier --filter-pattern reported and /aws/lambda/ci-runner-sweeper (see docs/runbook.md for the logs).

Acceptance criteria

  • A real reclaim is linked here (log line plus the check run), with the right commit and job name, or a bug filed if it was wrong.
  • A real 2-hour sweeper stop is linked here (log line plus the registration removed), or a bug filed if it failed.
  • Any new trap goes in docs/traps.md.

Blocked by

None.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions