Skip to content

Sweeper, notifier and swap repair for tasks #30

Closed

Description

@sodre

Parent

#18

What to build

Fargate tasks get the same lifetime rules and swap repair as instances.

  • Sweeper: it lists the cluster's tasks and stops any task 2 hours after launch. It counts a runner as having its machine if either an instance or a task carries its runner tag. Its role gains ecs:ListTasks, ecs:DescribeTasks, and ecs:StopTask on the stack's cluster only.
  • Notifier: a second rule matches ECS Task State Change on the stack's cluster with stopCode: SpotInterruption. It produces the same failed check run as an EC2 warning, from the task's tags.
  • Webhook: swap repair re-tags a task, not only an instance.

Acceptance criteria

  • Sweeper tests: a task over 2 hours is stopped; a registration is kept while its task lives and removed once it's gone; instances and tasks mixed.
  • Notifier tests: the ECS event and the EC2 warning produce identical check runs; a task without job tags is ignored.
  • Webhook tests: a swap involving a task re-queues the original job and re-tags the task.
  • The appJwt/gh copies stay identical, so the conventions test passes.
  • docs/stack.md and the runbook are updated.

Activity

  1. added a commit that references this issue on Nov 14, 2026
  2. sodre commented on Nov 14, 2026

    @sodre
    MemberAuthor

    Live (c6a2d74): four concurrent ci-fargate-spot jobs produced four {"swap","task"} repairs and all completed; re-tags landed under the aws:TagKeys condition. Sweeper ran clean over running tasks (ListTasks under ecs:cluster) and removed offline registrations once their tasks were gone; a real StopTask on a >2 h task not exercised (IAM simulated). Notifier: synthetic SpotInterruption events for a running and a stopped task created ci-runner / j (4) / j (1) with the real job names; a repeat logged already reported. A stopping/stopped task keeps its tags in describe-tasks --include TAGS (list-tags-for-resource refuses stopped tasks); docs corrected in 6dbaa56.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ready-for-agentFully specified, ready for an AFK agent

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions