Skip to content

Fargate spike: go/no-go #28

Closed
Closed

Description

@sodre

Parent

#18

What to build

A throwaway spike that answers #18's gate: is a Fargate Spot runner faster to start than EC2's ~100 s, or cheaper per job?

  • A script on a prototype/fargate-spot branch, not merged, hand-runs a real Rust build job from a consuming repo on Fargate Spot, at 4 vCPU / 16 GB, in x64 and in arm64.
  • The image is the ECR one, with its SOCI index. The task has an execution role and no task role. The runner gets a JIT config the way make hand-launch mints one.
  • The script creates whatever temporary resources it needs and deletes them afterwards.

Record, for each architecture:

  • queued-to-started time, from GitHub's job timestamps;
  • the task's pull start and stop times, and its start time;
  • the build's wall-clock time against the same job on ci-spot at ci-cpu-4;
  • cost per job, from the measured times and the Fargate Spot and EC2 Spot prices.

Acceptance criteria

  • The numbers above, for x64 and arm64, are recorded on this ticket and in the runbook's "Measured numbers".
  • A one-paragraph recommendation (go or no-go) against the gate: faster than ~100 s, or cheaper per job.
  • The branch is pushed, and this ticket links to it.
  • Every temporary resource is deleted.
  • The owner records the go/no-go decision on this ticket. On a no-go, write down why and stop here: close "Fargate capacities launch" and its dependants as wontfix.

Activity

  1. sodre commented on Nov 14, 2026

    @sodre
    MemberAuthor

    Spike results

    Branch: prototype/fargate-spot (ea4596a), script scripts/prototype-fargate-spike.sh (throwaway, not to be merged). The same numbers are in the runbook's "Measured numbers" on that branch.

    Setup. One job, a cold cargo build --locked -p ruff (ruff 0.17.5, debug, no cache) after dtolnay/rust-toolchain and cargo fetch, from a throwaway branch of ci-runner-canary. It's a public Rust project, not zae-zcm (the only Rust repo in the runner group): checking a private repo out from the canary would have needed a stored token. Fargate: a temporary FARGATE_SPOT cluster, 4 vCPU / 16 GB / 50 GB ephemeral, the ECR :latest image with its SOCI v2 index, an execution role and no task role, the runner as the container's command (./run.sh --jitconfig <cfg>, a JIT config minted as hand-launch.sh does), launched right after the push. EC2: during the first run, other repos' jobs held the stack's Spot budget (the launcher logged deferred: 40+4 of 24 spot vCPUs), so the EC2 runners were hand-launched with hand-launch.sh (the launcher's templates and fleet, capacity-optimized) under a private label.

    Fargate Spot x64 EC2 Spot x64 Fargate Spot arm64 EC2 Spot arm64
    CPU Xeon 8259CL (Cascade Lake) m6a.xlarge (EPYC 7R13), us-east-1b Graviton3 (Neoverse-V1) m8g.xlarge (Graviton4), us-east-1a
    Queued → started (GitHub) 27 s 102 s 30 s 92 s
    task createdAt 02:33:09.0 02:33:10.6
    pullStartedAt 02:33:28.4 (+19.3 s) 02:33:31.1 (+20.6 s)
    pullStoppedAt 02:33:31.1 (pull 2.8 s) 02:33:33.7 (pull 2.6 s)
    startedAt 02:33:30.9 02:33:33.5
    Listening for Jobs → job started 02:33:35 → 02:33:36 02:33:38 → 02:33:39
    Build step wall-clock 248 s 171 s 168 s 115 s
    Queued → completed 316 s 306 s 227 s 230 s
    Billed time 321 s (pull start → STOPPED) ~316 s (launch → job end + 15 s) 234 s ~234 s
    Price per hour $0.0879 (4 vCPU + 16 GB + 30 GB extra storage) $0.0700 Spot + $0.0055 gp3 $0.0710 $0.0926 Spot + $0.0055 gp3
    Cost per job $0.0078 $0.0066 $0.0046 $0.0064

    The first run (ripgrep, a 22 s build) gave the same picture: Fargate queued → started 22 s (x64) and 57 s (arm64). In the 57 s, the arm64 runner logged Listening for Jobs 33 s before GitHub gave it the job. Pulls took 2.3 s and 2.5 s, and provisioning (createdAt → pullStartedAt) took 13 to 14 s.

    Notes on the runner image on Fargate

    • The runner works directly as the container's command. The image has WORKDIR /home/runner, USER runner and no ENTRYPOINT, so ./run.sh --jitconfig runs with no wrapper. When the job ends, the runner exits 0 and the task stops (EssentialContainerExited), reaching STOPPED 24 to 29 s after the job completes.
    • The EC2 start script's Docker-host workarounds aren't needed: the copy of /home/runner out of the image, and the bind mounts of /tmp and the home at the same paths. On Fargate, /home/runner, _work and /tmp are the image's own paths on the 50 GB overlay (df: 51G). There is no /var/run/docker.sock, as ADR-0001 expects (the docker CLI is still in the image).
    • SOCI works: an 820 MB image "pulls" in about 3 s. A Fargate start is mostly the provisioning before the pull (13 to 21 s).
    • Fargate's CPUs are older than the EC2 families': x64 got Cascade Lake once and Sapphire Rapids once, and arm64 got Graviton3. The same build ran about 45% longer on both architectures.
    • In 3 of the 4 tasks, nproc said 8 and free said 30 GB on a 4 vCPU / 16 GB task, and there is no cgroup v2 cpu.max. A job that sizes its parallelism by nproc (cargo does) oversubscribes. The launch path could set CARGO_BUILD_JOBS/-j from the size label.
    • In the first run, ripgrep's integration tests failed on Fargate with NotFound opening a file. I didn't investigate; the second run measures only the build.

    Recommendation (for the owner's decision): go. The gate is "faster than ~100 s queued → started, or cheaper per job", and Fargate Spot passes the first test by a wide margin: 27 to 30 s against 92 to 102 s (one 57 s outlier was GitHub's own assignment delay). SOCI cuts the image pull to about 3 s. On cost it's a wash: arm64 was cheaper per job ($0.0046 against $0.0064 at today's m8g Spot price) and x64 dearer ($0.0078 against $0.0066 on m6a). The catch is CPU. Fargate's older hardware made this ~3 to 4-minute build about 45% slower, which cancels the faster start: queued → completed came out even (316 s against 306 s, 227 s against 230 s). Fargate wins outright for short Docker-free jobs, and loses for builds much longer than ~4 minutes. Job authors should get that trade-off in the README next to the boot times. The Fargate path should also cap build parallelism to the task's vCPUs, since nproc over-reports.

    All temporary resources are deleted: the cluster (INACTIVE), the task definitions (deregistered and deleted), the PowerUserPB-ci-runner-spike-exec role and its policy, the /ecs/ci-runner-spike-28 log group, the canary branch spike-28-fargate, the hand-launched instances (terminated), and the JIT runners (none left registered).

  2. sodre commented on Nov 14, 2026

    @sodre
    MemberAuthor

    Owner decision (2026-11-14): go. Build #29–#31. Notes for #29 from the spike: ./run.sh --jitconfig works directly as the container command (no host steps needed); nproc showed 8 CPUs on a 4-vCPU task in 3 of 4 tasks, so the Fargate path should cap job parallelism to the task's vCPUs; SOCI pull ~3 s. Prototype kept on prototype/fargate-spot (ea4596a).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    ready-for-agentFully specified, ready for an AFK agent

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions