Skip to content

Spec: shapes on every capacity, Fargate capacities, vCPU budgets and swap repair #18

Closed
13 of 14 issues completed

Description

@sodre

Problem Statement

Every CI runner today has the same shape: a 4-vCPU, 16 GB x64 EC2 instance, Spot or On-Demand, with a Docker daemon. A job author whose job doesn't fit that shape has no choice:

  • A Rust build that wants 8 or 16 vCPUs gets 4.
  • A small job that would fit in 2 vCPUs pays for 4.
  • A job that would be cheaper on Graviton can't ask for ARM.
  • A job that never uses Docker can't use Fargate, which needs no instance to boot, no host to clean up, and has its own Spot pool and quota.

There is also a live bug. GitHub gives a queued job to any idle runner whose labels cover the job's, so a runner often takes a job other than the one it was launched for (a swap). After a swap, the launcher can drop the other job's message as "not queued", and that job then waits for a runner that never comes. A Spot reclaim after a swap is also reported on the wrong commit.

Solution

Every job now names its full shape in runs-on, and the stack serves every shape on every capacity. The label contract becomes:

  • self-hosted
  • exactly one capacity label:
    • ci-spot: EC2 Spot
    • ci-ondemand: EC2 On-Demand
    • ci-fargate-spot: Fargate Spot
    • ci-fargate: Fargate On-Demand
  • exactly one size label, at 4 GB of memory per vCPU: ci-cpu-2, ci-cpu-4, ci-cpu-8 or ci-cpu-16
  • exactly one architecture label: x64 or arm64
  • linux, optionally

Nothing else is allowed, and none of the three has a default. Requiring all of them means a swap can only happen between runners of the same shape, which are interchangeable.

A job on a Fargate capacity has no Docker daemon. Picking one is the job author's promise not to use Docker. A job that uses it anyway fails at that step (ADR-0001).

Caps become vCPU budgets, one per capacity and shared by both architectures. The defaults equal today's caps at 4 vCPU per runner:

Budget Default
SpotVcpuCap 24
OnDemandVcpuCap 8
FargateSpotVcpuCap 24
FargateVcpuCap 8

Swaps are tolerated and repaired (ADR-0003). When a runner starts a job, the webhook checks whether that runner was launched for a different job. If so, it re-queues that other job and re-tags the runner with the job it really runs.

The Fargate path is gated. A hand-run spike measures queued-to-started time and cost per job on Fargate Spot, using the runner image from ECR with a lazy-loading index. The Fargate launch path is built only if the spike is faster than EC2's ~100 s boot, or cheaper per job. Everything else in this spec (shape labels on EC2, vCPU budgets, the swap repair, the multi-arch image) ships regardless.

User Stories

  1. As a job author, I want to choose 2, 4, 8 or 16 vCPUs for a job, so that a Rust build gets the CPU it needs and a small job doesn't pay for CPU it doesn't use.
  2. As a job author, I want every size to come with 4 GB of memory per vCPU, so that I can predict memory from the label alone.
  3. As a job author, I want to choose x64 or arm64 for a job, so that I can run on Graviton when my build supports it.
  4. As a job author, I want the same size and architecture labels to work on every capacity, so that switching between ci-spot and ci-fargate-spot is a one-label change that never strands a job.
  5. As a job author, I want a job with a missing or extra label to be ignored by the stack, as today, so that the label contract stays strict and easy to check.
  6. As a job author, I want the README to show complete runs-on examples for every capacity, so that I don't have to piece the label contract together.
  7. As a job author, I want ci-fargate-spot for Docker-free jobs, so that they run on Fargate Spot pricing without an EC2 boot.
  8. As a job author, I want ci-fargate for Docker-free jobs that must not be cut off half-way, so that I have an On-Demand choice on Fargate too.
  9. As a job author, I want a job on a Fargate capacity to fail at the step that uses Docker, so that the error points at the cause.
  10. As a job author, I want 50 GB of disk on every shape and capacity, so that a Rust target/ directory fits as it does today.
  11. As a job author, I want a job on Fargate to find no AWS credentials, as on EC2, so that a job's AWS access stays GitHub OIDC into the repo's own role.
  12. As a job author, I want the same toolset on Fargate as on EC2, from the same image, so that the repo's own toolset check passes on both.
  13. As a job author, I want a job on Fargate to have the same paths (/home/runner, _work, $RUNNER_TEMP, /tmp) as on EC2, so that scripts don't branch on capacity.
  14. As a job author, I want a Fargate Spot reclaim to produce the same failed check run, ci-runner / <job name> titled Spot capacity reclaimed: re-run this job, so that I recognise it the same way.
  15. As a job author, I want a reclaim to be reported on the commit of the job that was actually lost, even after a swap, so that I re-run the right job.
  16. As a job author, I want every queued job to get a runner even when runners swap jobs, so that no job waits forever for a runner.
  17. As a job author, I want a job whose capacity is at its vCPU budget to wait on the queue and start later, as today, so that a busy hour delays jobs instead of failing them.
  18. As an org owner, I want caps expressed as vCPU budgets, so that one 16-vCPU job and four 4-vCPU jobs count the same, matching how AWS quotas count.
  19. As an org owner, I want the budget parameters renamed (SpotVcpuCap and so on), so that nobody confuses the new meaning with the old runner counts.
  20. As an org owner, I want each capacity's budget to cover both architectures, so that there are four budgets to manage, not eight.
  21. As an org owner, I want the default budgets to equal today's caps at 4 vCPU per runner, so that a deploy doesn't change how much capacity the stack may use.
  22. As an org owner, I want the runbook to say which AWS quota each budget draws on (EC2 Spot, EC2 On-Demand, Fargate Spot, Fargate On-Demand vCPUs) and how to check it, so that I raise a budget only when the quota covers it.
  23. As an org owner, I want Fargate tasks to stop 2 hours after launch, as instances do, so that a hung task can't run up a bill.
  24. As an org owner, I want a Fargate task to exit after 15 minutes without a job, as an instance does, so that a runner whose job went elsewhere doesn't sit idle.
  25. As an org owner, I want the sweeper to remove a registration whose task is gone, as it does for instances, so that the runner group doesn't fill with offline runners.
  26. As an org owner, I want a Fargate task's runner output (Listening for Jobs, Running job) kept in CloudWatch for 7 days, so that I can see why a task never picked up a job.
  27. As an org owner, I want the runner image pushed to an ECR repo the stack owns, keeping the last 4 images, so that Fargate pulls it privately with lazy loading and old images don't pile up.
  28. As an org owner, I want the image push to use GitHub OIDC limited to zeroae/ci-runner's main, so that no long-lived AWS key or GitHub token is stored anywhere (ADR-0002).
  29. As an org owner, I want the stack to create GitHub's OIDC provider only when the account doesn't already have one, so that the deploy works in an account that already has it.
  30. As an org owner, I want EC2 runners to keep pulling from GHCR, so that instances still need no instance profile.
  31. As an org owner, I want the image built natively for amd64 and arm64, so that both architectures get the same tools, smoke-tested on their own hardware.
  32. As an org owner, I want arm64 EC2 runners drawn from m7g, m6g and m8g, and x64 from m7i, m6i, m6a and m7a, so that Spot capacity-optimized has several pools to choose from.
  33. As an org owner, I want make hand-launch to take a shape, so that I can hand-launch any size and architecture to test it.
  34. As an org owner, I want a hand-run Fargate spike that records queued-to-started time, the image pull time and the build time of a real Rust job, so that the go/no-go on the Fargate launch path rests on numbers.
  35. As an org owner, I want the measured Fargate numbers in the runbook's "Measured numbers", next to EC2's, so that job authors can choose a capacity knowing the boot time.
  36. As an org owner, I want the launcher's log to say which shape and capacity it launched, or deferred and at what budget, so that a "Waiting for a runner" is easy to diagnose.
  37. As an org owner, I want the launcher's log to record each swap it repairs (which runner took which job, and which job was re-queued), so that I can see how often swaps happen.
  38. As an org owner, I want the traps table to cover the new failure modes (a missing size or architecture label, Docker used on Fargate, a Fargate quota at its default of 6 vCPUs), so that the usual mistakes have a one-line answer.
  39. As an org owner, I want the stack to keep deploying as one stack, through an S3 bucket once the template outgrows 51,200 bytes, so that adding Fargate doesn't split the stack.
  40. As an org owner, I want the webhook to keep refusing public repos and repos outside zeroae, on every capacity, so that the fence holds for Fargate too.
  41. As an org owner, I want a duplicate delivery to launch at most one extra runner, as today, which the idle exit and the sweeper remove, so that duplicates stay harmless on Fargate too.
  42. As a consumer repo moving jobs, I want my existing two-label runs-on (self-hosted, ci-spot) to be called out in the release notes as needing a size and architecture label, so that I update it before the new label contract deploys.

Implementation Decisions

The label contract (webhook)

  • The webhook parses a job's labels, case-insensitively, into {capacity, vcpus, arch}. It enqueues that alongside today's fields, and enqueues nothing for a job outside the contract.
  • The contract: self-hosted; exactly one capacity label; exactly one size label; exactly one architecture label; optionally linux. No defaults. x64 and arm64 match GitHub's own runner labels, since labels are matched case-insensitively.
  • The fence (private repos in GITHUB_ORG only) is unchanged and runs before the contract.

Swap repair (webhook)

  • The webhook also handles workflow_job with action in_progress. It reads runner_name. If the name is one of the stack's (${stack}-<stamp>-<job id>) and its job id differs from the event's job:
    1. It re-queues the job the runner was launched for, with the fields it needs. The launcher's existing "still queued?" check then launches it a runner or drops it.
    2. It re-tags that runner's instance or task with the event's job: repo, job id, run id, head sha and job name.
  • The webhook's role gains ec2:DescribeInstances, ecs:ListTasks/ecs:DescribeTasks (to find the runner's instance or task), ec2:CreateTags and ecs:TagResource. Each is limited to resources carrying the stack's tag wherever IAM allows it.
  • To re-queue, the webhook needs the original job's message fields. It recovers them from the runner's own tags (repo, job id, run id, head sha, job name, and the shape labels), so nothing new is stored.
  • A runner taking its own job is the normal case and does nothing.

vCPU budgets (launcher)

  • The four runner-count parameters are replaced by SpotVcpuCap, OnDemandVcpuCap, FargateSpotVcpuCap and FargateVcpuCap, defaulting to 24, 8, 24 and 8. That's a breaking parameter rename; the runbook's override examples change with it.
  • The launcher sums the vCPUs of the capacity's live runners (instances pending/running/stopping/stopped, and tasks not yet STOPPED). If adding the job's vCPUs would exceed the budget, it re-queues the message with a 60 s delay, as today. The budget stays soft.
  • Each runner is tagged with its shape (ci-runner:capacity, ci-runner:vcpus, ci-runner:arch) so the budget can be counted without decoding instance types.

EC2 shapes (launcher, template)

  • The size label maps to .large, .xlarge, .2xlarge or .4xlarge. The architecture picks the family set: x64 uses m7i/m6i/m6a/m7a, and arm64 uses m7g/m6g/m8g. The fleet overrides list every family at that size in each of the three subnets, as today.
  • arm64 needs the AL2023 arm64 image. The template adds a second launch template, the arm64 counterpart of today's, through resolve:ssm: on /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-arm64. Otherwise it's the same: 50 GB gp3, IMDSv2, no instance profile.
  • make hand-launch takes the shape (capacity, size, architecture) as variables.

The image

  • image.yml builds amd64 and arm64 natively, on ubuntu-latest and ubuntu-24.04-arm; the repo is public, so both are free. It smoke-tests each and merges them into one multi-arch manifest.
  • On main, it pushes to GHCR as today, and to the stack's ECR repo through GitHub OIDC, with a SOCI v2 index made by soci convert before the push. Tags are :latest and :<date>-<sha>.
  • The stack owns: the ECR repo, with a lifecycle rule keeping the last 4 images; GitHub's OIDC provider, created only when the ExistingOidcProviderArn parameter is empty (foresight's github-deploy-zae pattern); and an image-push role PowerUserPB-${stack}-image-push. That role is trusted only for repo:zeroae/ci-runner:ref:refs/heads/main and allowed only to push to that repo.

Fargate (gated on the spike)

  • One ECS cluster in the stack, with the FARGATE and FARGATE_SPOT capacity providers.
  • Two task definitions, x64 and arm64, because the runtime platform can't be overridden. Each has:
    • the ECR image;
    • an execution role PowerUserPB-${stack}-runner-exec, allowed only to pull from the stack's repo and write to the log group;
    • no task role;
    • 50 GB of ephemeral storage;
    • awslogs to /ecs/${stack}-runner, kept 7 days.
  • The launcher calls RunTask with:
    • the capacity provider for the capacity;
    • task-level cpu/memory overrides for the size;
    • the JIT config in a container override (readable to ecs:DescribeTasks, accepted as the same exposure as EC2 user-data);
    • public IP on, in the three subnets with today's security group;
    • the same ci-runner:* tags as instances, including the stack's.
  • arm64 tasks skip any subnet in use1-az3, where Fargate offers no ARM.
  • The launcher's role gains ecs:RunTask on the two task definitions, iam:PassRole on the execution role only, and ecs:TagResource while creating.
  • The idle exit moves into the container. The image gets an entrypoint wrapper that runs run.sh --jitconfig and stops it after 15 minutes if the log has no Running job line. On EC2 the host script keeps doing it and the wrapper isn't used.
  • The sweeper also lists the cluster's tasks. It stops tasks 2 hours after launch, and it treats a runner as having its machine if a task or an instance carries its runner tag. Its role gains ecs:ListTasks, ecs:DescribeTasks and ecs:StopTask, the last limited to the stack's cluster.
  • The notifier gains a second EventBridge rule: ECS Task State Change on the stack's cluster with stopCode: SpotInterruption. Both rules produce the same check run from the runner's tags. Its role gains ecs:DescribeTasks.

Build and deploy

  • The template is at 49,908 of 51,200 bytes, so make deploy moves to the S3-bucket deploy (1 MB limit) before anything else grows. The bucket is created outside the stack, and the runbook says how.

Order

  1. Gate:
    • check the EC2 and Fargate vCPU quotas (Fargate Spot L-36FBB829 and Fargate On-Demand L-3032A538 default to 6) and request increases;
    • check that the permission set can create the OIDC provider, and whether zeroae-code already has one;
    • move to the S3-bucket deploy.
  2. The swap repair. It's independent and fixes a live bug, so it goes first.
  3. Two tracks in parallel:
    • the new label contract on EC2 (sizes, arm64), then vCPU budgets;
    • the multi-arch image, then the ECR push with OIDC and SOCI, then the Fargate spike, then go/no-go.
  4. Only if it's a go: the cluster and task definitions, the launcher's Fargate path, the in-container idle exit, then the sweeper and notifier for tasks, then a Rust canary.

Testing Decisions

  • Tests check external behaviour at the existing seam: each function's createHandler(deps), called with fake adapters as test/<name>.test.js does today. They assert what the handler sends to the outside: the queue message, the EC2 or ECS calls and their arguments, the GitHub calls, the log lines. They never assert internal helpers.
  • The webhook: signed deliveries in, queue messages and EC2/ECS tag calls out. Cases:
    • every valid shape on every capacity;
    • each way to break the label contract: a missing, duplicate or unknown size label; a missing or duplicate architecture label; an unknown label;
    • in_progress on a runner's own job, which does nothing;
    • in_progress after a swap, which re-queues the original job from the runner's tags and re-tags the runner, for both an instance and a task;
    • in_progress on a runner that isn't the stack's, which is ignored.
  • The launcher:
    • the fleet overrides for every size and architecture;
    • vCPU budgets, summed across instances and tasks of mixed sizes;
    • deferral at the budget;
    • the RunTask call (capacity provider, overrides, tags, subnets for arm64);
    • removing the registration when no instance or task launches.
  • The sweeper: tasks over 2 hours stopped; a registration kept for a live task and removed when the task is gone; a mix of instances and tasks.
  • The notifier: an ECS SpotInterruption event and an EC2 warning produce the same check run; a task with no job tags is ignored.
  • The image: image/smoke.sh runs on both architectures in image.yml, and also checks the entrypoint wrapper's idle exit with a short timeout.
  • Template checks: test/build.test.js and test/conventions.test.js stay in force (template size against the deploy limit, identical appJwt/gh copies). The size limit becomes the S3 deploy's.
  • Fakes or the emulator: fakes for unit tests, as today. Where a test needs real AWS API shapes, such as ECS RunTask/DescribeTasks responses or the event formats, use aws/aws-emulator rather than hand-written guesses.
  • Prior art: test/launcher.test.js (fake ec2, sqs, fetch), test/sweeper.test.js, test/notifier.test.js, test/webhook.test.js.
  • Live proof: the Fargate spike and the Rust canary are the live tests, as Canary proof in a private repo #12 was for EC2. The canary shows, on our runners:
    • a Rust job on ci-fargate-spot and one on ci-spot at ci-cpu-8/x64;
    • a job on arm64;
    • a swap repaired, with no stranded job;
    • a Fargate Spot reclaim producing its check run on the right commit;
    • a job deferred at a vCPU budget;
    • a task stopped by the sweeper.

Out of Scope

  • A warm pool. Nothing is started before its job queues.
  • Docker on the Fargate capacities, whether rootless, sidecar or remote (ADR-0001).
  • A run label or per-job label to prevent swaps (ADR-0003).
  • Disk sizes other than 50 GB, and memory ratios other than 4 GB per vCPU.
  • Sizes above ci-cpu-16, GPUs, macOS and Windows (macOS and Windows runners #17).
  • Per-repo budgets.
  • EC2 runners pulling from ECR.
  • Pinning the image, the runner or the tools.
  • Automatic retry of a job lost to a reclaim.
  • A Rust build cache (sccache or similar) or a warm toolcache.

Further Notes

  • Spike inputs, us-east-1, from the AWS price feeds of 2026-11-11:

    Capacity Shape Price per hour
    Fargate Spot x64 4 vCPU / 16 GB about $0.085
    Fargate Spot arm64 4 vCPU / 16 GB about $0.068
    EC2 Spot m7i.xlarge (x64) 4 vCPU / 16 GB about $0.090, excluding EBS
    EC2 Spot m7g.xlarge (arm64) 4 vCPU / 16 GB about $0.068; the feed may be stale

    At equal size, cost per job is roughly a wash, so the gate likely turns on boot time. The image is about 820 MB compressed. Fargate caches nothing, and SOCI only lazy-loads from ECR. One published measurement in us-east-1 went from 35 s to 17–21 s with a v2 index.

  • Unverified: whether AWSPowerUserAccess allows iam:CreateOpenIDConnectProvider; whether zeroae-code already has GitHub's provider; the account's current Fargate quotas. The SSO session had expired when this was researched. All three belong to the gate.

  • Decisions: ADR-0001 (Fargate capacities have no Docker), ADR-0002 (push the image to ECR from CI), ADR-0003 (tolerate swaps instead of run labels). Terms are in GLOSSARY.md.

  • Rollout: the new label contract is a breaking change. A job with today's runs-on (self-hosted, ci-spot) falls outside it on the day it deploys, and waits forever. Before that ticket deploys, every repo already moved (ci-runner-canary and the first consumer) must add ci-cpu-4 and x64. Adding them is harmless under today's contract only if the webhook already allows them, so the label ticket ships in two steps: first allow the new labels, then require them.

  • This replaces the deferred "Fargate capacity (ci-fargate)" and "More instance sizes" items in the 2026-11-12 design spec.

  • The swap fix comes from CodeBuild-hosted runners' own documented failure: "a job with fewer labels can be picked up by a runner created for a job with more labels".

Activity

  1. sodre commented on Nov 14, 2026

    @sodre
    MemberAuthor

    Amendment (2026-11-14, owner decision): FargateSpotVcpuCap defaults to 48, not 24 (supersedes the budget table and story 21 for Fargate Spot). At 24, a consumer's 9-job ci-cpu-8 matrix spent most of its queued time on budget waits; the Fargate Spot quota is 140. The other budgets stay 24 / 8 / 8. Recorded in docs/stack.md (commit 097cd81).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ready-for-agentFully specified, ready for an AFK agent

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions