You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
A throwaway spike that answers #18's gate: is a Fargate Spot runner faster to start than EC2's ~100 s, or cheaper per job?
A script on a prototype/fargate-spot branch, not merged, hand-runs a real Rust build job from a consuming repo on Fargate Spot, at 4 vCPU / 16 GB, in x64 and in arm64.
The image is the ECR one, with its SOCI index. The task has an execution role and no task role. The runner gets a JIT config the way make hand-launch mints one.
The script creates whatever temporary resources it needs and deletes them afterwards.
Record, for each architecture:
queued-to-started time, from GitHub's job timestamps;
the task's pull start and stop times, and its start time;
the build's wall-clock time against the same job on ci-spot at ci-cpu-4;
cost per job, from the measured times and the Fargate Spot and EC2 Spot prices.
Acceptance criteria
The numbers above, for x64 and arm64, are recorded on this ticket and in the runbook's "Measured numbers".
A one-paragraph recommendation (go or no-go) against the gate: faster than ~100 s, or cheaper per job.
The branch is pushed, and this ticket links to it.
Every temporary resource is deleted.
The owner records the go/no-go decision on this ticket. On a no-go, write down why and stop here: close "Fargate capacities launch" and its dependants as wontfix.
Branch: prototype/fargate-spot (ea4596a), script scripts/prototype-fargate-spike.sh (throwaway, not to be merged). The same numbers are in the runbook's "Measured numbers" on that branch.
Setup. One job, a cold cargo build --locked -p ruff (ruff 0.17.5, debug, no cache) after dtolnay/rust-toolchain and cargo fetch, from a throwaway branch of ci-runner-canary. It's a public Rust project, not zae-zcm (the only Rust repo in the runner group): checking a private repo out from the canary would have needed a stored token. Fargate: a temporary FARGATE_SPOT cluster, 4 vCPU / 16 GB / 50 GB ephemeral, the ECR :latest image with its SOCI v2 index, an execution role and no task role, the runner as the container's command (./run.sh --jitconfig <cfg>, a JIT config minted as hand-launch.sh does), launched right after the push. EC2: during the first run, other repos' jobs held the stack's Spot budget (the launcher logged deferred: 40+4 of 24 spot vCPUs), so the EC2 runners were hand-launched with hand-launch.sh (the launcher's templates and fleet, capacity-optimized) under a private label.
Fargate Spot x64
EC2 Spot x64
Fargate Spot arm64
EC2 Spot arm64
CPU
Xeon 8259CL (Cascade Lake)
m6a.xlarge (EPYC 7R13), us-east-1b
Graviton3 (Neoverse-V1)
m8g.xlarge (Graviton4), us-east-1a
Queued → started (GitHub)
27 s
102 s
30 s
92 s
task createdAt
02:33:09.0
02:33:10.6
pullStartedAt
02:33:28.4 (+19.3 s)
02:33:31.1 (+20.6 s)
pullStoppedAt
02:33:31.1 (pull 2.8 s)
02:33:33.7 (pull 2.6 s)
startedAt
02:33:30.9
02:33:33.5
Listening for Jobs → job started
02:33:35 → 02:33:36
02:33:38 → 02:33:39
Build step wall-clock
248 s
171 s
168 s
115 s
Queued → completed
316 s
306 s
227 s
230 s
Billed time
321 s (pull start → STOPPED)
~316 s (launch → job end + 15 s)
234 s
~234 s
Price per hour
$0.0879 (4 vCPU + 16 GB + 30 GB extra storage)
$0.0700 Spot + $0.0055 gp3
$0.0710
$0.0926 Spot + $0.0055 gp3
Cost per job
$0.0078
$0.0066
$0.0046
$0.0064
The first run (ripgrep, a 22 s build) gave the same picture: Fargate queued → started 22 s (x64) and 57 s (arm64). In the 57 s, the arm64 runner logged Listening for Jobs 33 s before GitHub gave it the job. Pulls took 2.3 s and 2.5 s, and provisioning (createdAt → pullStartedAt) took 13 to 14 s.
Notes on the runner image on Fargate
The runner works directly as the container's command. The image has WORKDIR /home/runner, USER runner and no ENTRYPOINT, so ./run.sh --jitconfig runs with no wrapper. When the job ends, the runner exits 0 and the task stops (EssentialContainerExited), reaching STOPPED 24 to 29 s after the job completes.
The EC2 start script's Docker-host workarounds aren't needed: the copy of /home/runner out of the image, and the bind mounts of /tmp and the home at the same paths. On Fargate, /home/runner, _work and /tmp are the image's own paths on the 50 GB overlay (df: 51G). There is no /var/run/docker.sock, as ADR-0001 expects (the docker CLI is still in the image).
SOCI works: an 820 MB image "pulls" in about 3 s. A Fargate start is mostly the provisioning before the pull (13 to 21 s).
Fargate's CPUs are older than the EC2 families': x64 got Cascade Lake once and Sapphire Rapids once, and arm64 got Graviton3. The same build ran about 45% longer on both architectures.
In 3 of the 4 tasks, nproc said 8 and free said 30 GB on a 4 vCPU / 16 GB task, and there is no cgroup v2 cpu.max. A job that sizes its parallelism by nproc (cargo does) oversubscribes. The launch path could set CARGO_BUILD_JOBS/-j from the size label.
In the first run, ripgrep's integration tests failed on Fargate with NotFound opening a file. I didn't investigate; the second run measures only the build.
Recommendation (for the owner's decision): go. The gate is "faster than ~100 s queued → started, or cheaper per job", and Fargate Spot passes the first test by a wide margin: 27 to 30 s against 92 to 102 s (one 57 s outlier was GitHub's own assignment delay). SOCI cuts the image pull to about 3 s. On cost it's a wash: arm64 was cheaper per job ($0.0046 against $0.0064 at today's m8g Spot price) and x64 dearer ($0.0078 against $0.0066 on m6a). The catch is CPU. Fargate's older hardware made this ~3 to 4-minute build about 45% slower, which cancels the faster start: queued → completed came out even (316 s against 306 s, 227 s against 230 s). Fargate wins outright for short Docker-free jobs, and loses for builds much longer than ~4 minutes. Job authors should get that trade-off in the README next to the boot times. The Fargate path should also cap build parallelism to the task's vCPUs, since nproc over-reports.
All temporary resources are deleted: the cluster (INACTIVE), the task definitions (deregistered and deleted), the PowerUserPB-ci-runner-spike-exec role and its policy, the /ecs/ci-runner-spike-28 log group, the canary branch spike-28-fargate, the hand-launched instances (terminated), and the JIT runners (none left registered).
Owner decision (2026-11-14): go. Build #29–#31. Notes for #29 from the spike: ./run.sh --jitconfig works directly as the container command (no host steps needed); nproc showed 8 CPUs on a 4-vCPU task in 3 of 4 tasks, so the Fargate path should cap job parallelism to the task's vCPUs; SOCI pull ~3 s. Prototype kept on prototype/fargate-spot (ea4596a).
Activity
sodre commentedon Nov 14, 2026
Spike results
Branch:
prototype/fargate-spot(ea4596a), scriptscripts/prototype-fargate-spike.sh(throwaway, not to be merged). The same numbers are in the runbook's "Measured numbers" on that branch.Setup. One job, a cold
cargo build --locked -p ruff(ruff 0.17.5, debug, no cache) afterdtolnay/rust-toolchainandcargo fetch, from a throwaway branch ofci-runner-canary. It's a public Rust project, notzae-zcm(the only Rust repo in the runner group): checking a private repo out from the canary would have needed a stored token. Fargate: a temporary FARGATE_SPOT cluster, 4 vCPU / 16 GB / 50 GB ephemeral, the ECR:latestimage with its SOCI v2 index, an execution role and no task role, the runner as the container's command (./run.sh --jitconfig <cfg>, a JIT config minted ashand-launch.shdoes), launched right after the push. EC2: during the first run, other repos' jobs held the stack's Spot budget (the launcher loggeddeferred: 40+4 of 24 spot vCPUs), so the EC2 runners were hand-launched withhand-launch.sh(the launcher's templates and fleet,capacity-optimized) under a private label.m6a.xlarge(EPYC 7R13), us-east-1bm8g.xlarge(Graviton4), us-east-1acreatedAtpullStartedAtpullStoppedAtstartedAtListening for Jobs→ job startedSTOPPED)The first run (ripgrep, a 22 s build) gave the same picture: Fargate queued → started 22 s (x64) and 57 s (arm64). In the 57 s, the arm64 runner logged
Listening for Jobs33 s before GitHub gave it the job. Pulls took 2.3 s and 2.5 s, and provisioning (createdAt→pullStartedAt) took 13 to 14 s.Notes on the runner image on Fargate
WORKDIR /home/runner,USER runnerand noENTRYPOINT, so./run.sh --jitconfigruns with no wrapper. When the job ends, the runner exits 0 and the task stops (EssentialContainerExited), reachingSTOPPED24 to 29 s after the job completes./home/runnerout of the image, and the bind mounts of/tmpand the home at the same paths. On Fargate,/home/runner,_workand/tmpare the image's own paths on the 50 GB overlay (df: 51G). There is no/var/run/docker.sock, as ADR-0001 expects (thedockerCLI is still in the image).nprocsaid 8 andfreesaid 30 GB on a 4 vCPU / 16 GB task, and there is no cgroup v2cpu.max. A job that sizes its parallelism bynproc(cargo does) oversubscribes. The launch path could setCARGO_BUILD_JOBS/-jfrom the size label.NotFoundopening a file. I didn't investigate; the second run measures only the build.Recommendation (for the owner's decision): go. The gate is "faster than ~100 s queued → started, or cheaper per job", and Fargate Spot passes the first test by a wide margin: 27 to 30 s against 92 to 102 s (one 57 s outlier was GitHub's own assignment delay). SOCI cuts the image pull to about 3 s. On cost it's a wash: arm64 was cheaper per job ($0.0046 against $0.0064 at today's
m8gSpot price) and x64 dearer ($0.0078 against $0.0066 onm6a). The catch is CPU. Fargate's older hardware made this ~3 to 4-minute build about 45% slower, which cancels the faster start: queued → completed came out even (316 s against 306 s, 227 s against 230 s). Fargate wins outright for short Docker-free jobs, and loses for builds much longer than ~4 minutes. Job authors should get that trade-off in the README next to the boot times. The Fargate path should also cap build parallelism to the task's vCPUs, sincenprocover-reports.All temporary resources are deleted: the cluster (INACTIVE), the task definitions (deregistered and deleted), the
PowerUserPB-ci-runner-spike-execrole and its policy, the/ecs/ci-runner-spike-28log group, the canary branchspike-28-fargate, the hand-launched instances (terminated), and the JIT runners (none left registered).sodre commentedon Nov 14, 2026
Owner decision (2026-11-14): go. Build #29–#31. Notes for #29 from the spike:
./run.sh --jitconfigworks directly as the container command (no host steps needed);nprocshowed 8 CPUs on a 4-vCPU task in 3 of 4 tasks, so the Fargate path should cap job parallelism to the task's vCPUs; SOCI pull ~3 s. Prototype kept onprototype/fargate-spot(ea4596a).Runbook: the Fargate spike's measured numbers (#28)
Runbook: Rust mutation-test matrix numbers per shape; Fargate Spot bu…