Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
a9e7b48
Glossary and ADRs for shapes, Fargate capacities and swaps (#18)
sodre Nov 14, 2026
0767d3d
Smoke: check every added binary is built for the image's architecture
sodre Nov 14, 2026
fef784a
Build: check the template against the S3 deploy's 1 MB limit
sodre Nov 14, 2026
06b38d2
image.yml: build amd64 and arm64 natively, publish one manifest list
sodre Nov 14, 2026
2e22b86
make deploy and make validate go through the deploy bucket
sodre Nov 14, 2026
a19f130
Docs: the deploy bucket, how it's made, and the 1 MB limit
sodre Nov 14, 2026
9d9a85e
Merge origin/integration/fargate-capacities into ticket/23-multi-arch…
sodre Nov 14, 2026
7a2707b
Runbook: the four vCPU quotas each budget draws on (gate, #19)
sodre Nov 14, 2026
bf44d0e
Merge ticket #23: multi-arch runner image
sodre Nov 14, 2026
0b78a8a
Merge origin/integration/fargate-capacities into ticket/19-gate
sodre Nov 14, 2026
ab7ecd7
Webhook: accept the size labels and x64, and enqueue the job's vCPUs …
sodre Nov 14, 2026
8fcab6e
Merge ticket #19: gate (quotas, OIDC permission sets)
sodre Nov 14, 2026
585b1e2
Launcher: launch the job's size across the x64 families, and tag the …
sodre Nov 14, 2026
25919b5
make hand-launch takes a size (SIZE=ci-cpu-8)
sodre Nov 14, 2026
c831740
Docs: the size labels in the README, the stack doc and the runbook
sodre Nov 14, 2026
640f08a
Launcher: tag the instance with the job's labels
sodre Nov 14, 2026
ab08ae6
Webhook: repair swaps on workflow_job in_progress
sodre Nov 14, 2026
e38e978
Merge origin/integration/fargate-capacities into ticket/22-size-labels
sodre Nov 14, 2026
fe7dcfa
Merge ticket #22: size labels on EC2 (expand), live proof pending
sodre Nov 14, 2026
2d21d43
Docs: the webhook's swap repair, its log line and the swap trap
sodre Nov 14, 2026
ddf9eb7
Webhook: rewrap the header comment
sodre Nov 14, 2026
05952bb
Webhook: accept arm64 as an architecture label
sodre Nov 14, 2026
01b589e
Launcher: vCPU budgets replace the runner caps
sodre Nov 14, 2026
558bd8b
Docs: vCPU budgets in the stack, runbook and README
sodre Nov 14, 2026
fde691a
Launcher: launch arm64 jobs on m7g, m6g and m8g from the arm64 launch…
sodre Nov 14, 2026
6c75e29
Template: an arm64 launch template, and the launcher may launch from …
sodre Nov 14, 2026
4afdbc3
hand-launch: ARCH=arm64 launches a Graviton runner from the arm64 lau…
sodre Nov 14, 2026
c194221
Merge main (multi-arch image landed as #33, compilers #16) into integ…
sodre Nov 14, 2026
3c95576
Merge ticket #25: vCPU budgets, live proof pending
sodre Nov 14, 2026
9bd8db1
Docs: arm64 in the README, the stack doc and the runbook
sodre Nov 14, 2026
8f9d944
Merge origin/integration/fargate-capacities into ticket/24-arm64-ec2
sodre Nov 14, 2026
fcebb43
Merge ticket #24: arm64 on EC2, live proof pending
sodre Nov 14, 2026
2a9b1bc
Webhook: require a size and an architecture label, no defaults
sodre Nov 14, 2026
9e0c02c
image.yml: make a SOCI v2 index for both architectures, push it to EC…
sodre Nov 14, 2026
fa0be3e
hand-launch: require a size and an architecture
sodre Nov 14, 2026
356b55b
Docs: the full label contract, with no size or architecture default
sodre Nov 14, 2026
f103750
Stack: an ECR repo for the runner image, and an OIDC push role for main
sodre Nov 14, 2026
1032dc5
Deploy and operate under zeroae-code/ZAEPowerUserAccess by default
sodre Nov 14, 2026
72cde1c
Docs: the ECR repo, the push role and its setup, and their traps
sodre Nov 14, 2026
18b815e
Merge remote-tracking branch 'origin/integration/fargate-capacities' …
sodre Nov 14, 2026
7e539c3
Tests: the image push's template checks join the other template promises
sodre Nov 14, 2026
9e9b53a
Merge ticket #27: image to ECR through OIDC with a SOCI index, live p…
sodre Nov 14, 2026
d51a618
Merge origin/integration/fargate-capacities into ticket/21-swap-repair
sodre Nov 14, 2026
f3a7899
Merge ticket #21: swap repair for instances
sodre Nov 14, 2026
b28303f
Push role: trust the immutable OIDC subject
sodre Nov 14, 2026
bf15731
Runbook: a vCPU budget is soft, and a burst goes over it
sodre Nov 14, 2026
cabcc32
Merge remote-tracking branch 'origin/integration/fargate-capacities' …
sodre Nov 14, 2026
6b912aa
Runbook: the Fargate spike's measured numbers (#28)
sodre Nov 14, 2026
5ad5833
Image: an idle-exit entrypoint for the Fargate capacities
sodre Nov 14, 2026
c95fe2b
Webhook: accept the two Fargate capacity labels
sodre Nov 14, 2026
7e69020
Launcher and stack: one Fargate task per ci-fargate-spot or ci-fargat…
sodre Nov 14, 2026
019b851
CI: install aws-emulator for the ECS tests
sodre Nov 14, 2026
e05b7aa
Merge ticket #26: require the full label contract
sodre Nov 14, 2026
4565e46
Docs: the Fargate capacities
sodre Nov 14, 2026
8454a9c
Merge remote-tracking branch 'origin/integration/fargate-capacities' …
sodre Nov 14, 2026
5a6c7ed
Webhook test: a Fargate job needs its size and architecture too
sodre Nov 14, 2026
3d1d6e1
Merge ticket #29: Fargate capacities, live proof pending
sodre Nov 14, 2026
1e4a9bc
Sweeper: stop tasks over 2 hours old; a task keeps its runner's regis…
sodre Nov 14, 2026
d97ba42
Notifier: report a Fargate Spot reclaim, with the job's real name fro…
sodre Nov 14, 2026
2c1d465
Webhook: repair a swap on a task, re-tagged as the launcher tags it
sodre Nov 14, 2026
08ae417
Tests: share the aws-emulator plumbing
sodre Nov 14, 2026
59067c2
Launcher role: the ZAEPowerUserBoundary, so RunTask can pass the exec…
sodre Nov 14, 2026
68648e3
x64 runners also launch on m8i and m8a
sodre Nov 14, 2026
5eafcc6
Merge remote-tracking branch 'origin/integration/fargate-capacities' …
sodre Nov 14, 2026
6914755
Notifier: match every change of a Fargate Spot reclaim, report once p…
sodre Nov 14, 2026
d743d6c
Docs: the sweeper, notifier and swap repair for tasks
sodre Nov 14, 2026
8117edb
Merge remote-tracking branch 'origin/integration/fargate-capacities' …
sodre Nov 14, 2026
6206da7
Merge ticket #30: sweeper, notifier and swap repair for tasks, live p…
sodre Nov 14, 2026
032b5d8
Idle exit: stop the runner's whole session, and KILL it after a grace
sodre Nov 14, 2026
45ff766
Docs: what the Fargate live proof found
sodre Nov 14, 2026
f1bf750
Merge remote-tracking branch 'origin/integration/fargate-capacities' …
sodre Nov 14, 2026
c6a2d74
Merge ticket #29: Fargate capacities (PassRole boundary, idle-exit fix)
sodre Nov 14, 2026
6dbaa56
Docs: ECS keeps a stopped task's tags readable through DescribeTasks
sodre Nov 14, 2026
f155abc
Launcher: a task told to stop counts towards the Fargate budget until…
sodre Nov 14, 2026
9ec3e9c
Launcher, webhook: one named default shape per file
sodre Nov 14, 2026
b0c3959
Sweeper: derive the stop reason's age from MAX_AGE_MS; re-wrap the he…
sodre Nov 14, 2026
9f79ce2
Webhook, notifier: names that say what they hold
sodre Nov 14, 2026
c7cbdc6
Launcher, webhook, notifier: one tagsOf() per file for a tag list as …
sodre Nov 14, 2026
932a287
Template: NotifierTaskRule's comment says ECS keeps a stopped task's …
sodre Nov 14, 2026
89c1e92
stack.md: Adding a Lambda names the ZAEPowerUserBoundary exception
sodre Nov 14, 2026
00c9fb7
Glossary: narrow vCPU budget's 'avoid cap' to prose; the parameters a…
sodre Nov 14, 2026
ee01946
README: runs-on examples for ci-fargate-spot and ci-fargate, and the …
sodre Nov 14, 2026
824b2af
hand-launch: take a required CAPACITY, ci-spot or ci-ondemand
sodre Nov 14, 2026
c30ebf0
Merge remote-tracking branch 'origin/main' into integration/fargate-c…
sodre Nov 14, 2026
8cda39f
Image: the Fargate entrypoint caps cargo's jobs in ~/.cargo/config.toml
sodre Nov 14, 2026
cd38716
Launcher: CI_RUNNER_VCPUS for Fargate tasks, not CARGO_BUILD_JOBS
sodre Nov 14, 2026
097cd81
Fargate Spot budget defaults to 48 vCPUs
sodre Nov 14, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,23 @@ jobs:
- uses: actions/setup-node@v4
with:
node-version: 22
# aws/aws-emulator, for the tests that need real ECS shapes
# (test/launcher-ecs.test.js). The package is signed by AWS; check the
# signing key's fingerprint and the signature before installing it.
- name: Install aws-emulator
run: |
base=https://downloads.aws.com/aws-emulator/packages/aws-emulator/LTS/latest
curl -fsSLO "$base/aws-emulator_amd64.deb"
curl -fsSLO "$base/aws-emulator_amd64.deb.sig"
curl -fsSLo aws-publishing.asc https://downloads.aws.com/keys/gpg/current.asc
gpg --import aws-publishing.asc
gpg --fingerprint --with-colons ssm-publishing@amazon.com | grep -q '^fpr:::::::::98BCF336A3BE53B734E967C52E5BF49BD940F624:$'
gpg --verify aws-emulator_amd64.deb.sig aws-emulator_amd64.deb
sudo apt-get install -y ./aws-emulator_amd64.deb
aws-emulator -v
# The unit tests and the repo conventions, then the template build,
# which fails at or over the 51,200-byte inline limit.
# which fails at or over the S3 deploy's 1 MB limit.
- run: make test
env:
REQUIRE_AWS_EMULATOR: "1"
- run: make build
2 changes: 1 addition & 1 deletion CODING_STANDARDS.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ adoption variable `ZAE_CI_RUNNER`.
- **No dependencies.** Each function is one CommonJS file under `functions/`,
using only Node.js 22 built-ins and the `@aws-sdk` clients the runtime
ships. The build inlines it into the template, so every byte counts against
the 51,200-byte limit.
the 1 MB limit.
- **A `createHandler(deps)` factory** holds the logic and takes every side
effect (AWS clients, `fetch`, `now`, `env`, `log`) as a dependency, so the
tests run the real code with fakes. `exports.handler` only wires the real
Expand Down
43 changes: 43 additions & 0 deletions GLOSSARY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# ci-runner

Single-use self-hosted GitHub Actions runners for the private repos of the `zeroae` org.

## Language

**CI runner**:
A self-hosted runner created for one queued job. It runs that job and is then gone.
_Avoid_: agent, worker, pool member

**Capacity**:
The kind of compute a CI runner runs on: EC2 Spot, EC2 On-Demand, Fargate Spot or Fargate On-Demand.
_Avoid_: pool, warm pool (nothing is started before its job queues)

**Capacity label**:
The `runs-on` label that picks a job's capacity: `ci-spot`, `ci-ondemand`, `ci-fargate-spot` or `ci-fargate`.

**Size label**:
The `runs-on` label that picks a CI runner's vCPU count: `ci-cpu-2`, `ci-cpu-4`, `ci-cpu-8` or `ci-cpu-16`, with 4 GB of memory per vCPU. It has no default.
_Avoid_: instance type, xlarge, instance-size

**Architecture label**:
The `runs-on` label that picks a CI runner's CPU architecture: `x64` or `arm64`. It has no default.

**Shape**:
A CI runner's capacity, size and architecture together. Runners of the same shape are interchangeable.

**Label contract**:
The labels a job must carry to be served: `self-hosted`, exactly one capacity label, exactly one size label and exactly one architecture label, plus `linux` if the job wants it. Nothing else is allowed. It holds on every capacity.

**Swap**:
A CI runner taking a queued job other than the one it was launched for. GitHub gives a job to any idle runner whose labels cover the job's, so swaps between runners of the same shape are normal.

**Fargate capacity**:
Fargate Spot or Fargate On-Demand, the two capacities with no Docker daemon. A job that picks one promises not to use Docker.
_Avoid_: Fargate pool

**vCPU budget**:
The most vCPUs that the CI runners of one capacity may hold at once. A job that would go over it waits.
_Avoid_: "cap" in prose (that was the old runner count). The parameters that set the budgets are still named `*VcpuCap` (`SpotVcpuCap`, `OnDemandVcpuCap`, `FargateSpotVcpuCap`, `FargateVcpuCap`).

**Spot reclaim**:
AWS taking back Spot capacity, EC2 or Fargate, while a job runs. The job is lost and is reported as one on its commit.
20 changes: 12 additions & 8 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
STACK ?= ci-runner
PROFILE ?= zeroae-code/AWSPowerUserAccess
PROFILE ?= zeroae-code/ZAEPowerUserAccess
REGION ?= us-east-1
ORG ?= zeroae
AWS = aws --profile $(PROFILE) --region $(REGION)
# The deploy bucket, made outside the stack (docs/runbook.md). Looked up only when used.
BUCKET ?= $(STACK)-deploy-$(shell $(AWS) sts get-caller-identity --query Account --output text)

.PHONY: help test build validate deploy outputs hand-launch runners sweep canary-runs
.DEFAULT_GOAL := help
Expand All @@ -17,29 +19,31 @@ test: ## Run the unit tests (node --test, no npm deps)
build: ## Inline functions/*.js into build/template.yaml and report its size
node scripts/build-template.js

validate: build ## Ask CloudFormation to validate the built template
$(AWS) cloudformation validate-template --template-body file://build/template.yaml >/dev/null && echo valid
validate: build ## Ask CloudFormation to validate the built template (through the deploy bucket)
$(AWS) s3 cp --only-show-errors build/template.yaml s3://$(BUCKET)/$(STACK)/validate.yaml
$(AWS) cloudformation validate-template \
--template-url https://$(BUCKET).s3.$(REGION).amazonaws.com/$(STACK)/validate.yaml >/dev/null && echo valid

deploy: test build ## Create or update the stack ($(STACK)) from one command
deploy: test build ## Create or update the stack ($(STACK)) through the deploy bucket
$(AWS) cloudformation deploy --stack-name $(STACK) \
--template-file build/template.yaml \
--template-file build/template.yaml --s3-bucket $(BUCKET) --s3-prefix $(STACK) \
--capabilities CAPABILITY_NAMED_IAM --no-fail-on-empty-changeset
@$(MAKE) --no-print-directory outputs

outputs: ## Print the stack's outputs (setup URL, webhook URL, secret ARN)
@$(AWS) cloudformation describe-stacks --stack-name $(STACK) \
--query 'Stacks[0].Outputs[].[OutputKey,OutputValue]' --output text

hand-launch: ## Launch one CI runner by hand for one ci-spot job (needs gh with admin:org)
STACK=$(STACK) PROFILE=$(PROFILE) REGION=$(REGION) scripts/hand-launch.sh
hand-launch: ## Launch one EC2 CI runner by hand of shape CAPACITY (ci-spot or ci-ondemand), SIZE and ARCH, all required, e.g. CAPACITY=ci-spot SIZE=ci-cpu-4 ARCH=x64 (needs gh with admin:org)
STACK=$(STACK) PROFILE=$(PROFILE) REGION=$(REGION) CAPACITY=$(CAPACITY) SIZE=$(SIZE) ARCH=$(ARCH) scripts/hand-launch.sh

runners: ## List the stack's CI runner instances that are not terminated
@$(AWS) ec2 describe-instances \
--filters Name=tag:ci-runner:stack,Values=$(STACK) \
Name=instance-state-name,Values=pending,running,stopping,stopped,shutting-down \
--query 'Reservations[].Instances[].[InstanceId,State.Name,InstanceType,LaunchTime]' --output text

sweep: ## Run the sweeper now, and print what it terminated and removed
sweep: ## Run the sweeper now, and print what it terminated, stopped and removed
@$(AWS) lambda invoke --function-name $(STACK)-sweeper /dev/stdout --query FunctionError --output text

canary-runs: ## List the canary repo's recent workflow runs (CI_RUNNER_CANARY_REPO, from .env)
Expand Down
94 changes: 78 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,21 +1,48 @@
# ci-runner

Single-use EC2 runners for the GitHub Actions jobs of zeroae's private repos.
Single-use EC2 and Fargate runners for the GitHub Actions jobs of zeroae's
private repos.

Each queued job labelled `ci-spot` or `ci-ondemand` gets a fresh `.xlarge`
EC2 instance. The instance runs that one job in `ghcr.io/zeroae/ci-runner:latest`
Each queued job labelled `ci-spot` or `ci-ondemand` gets a fresh EC2
instance of the size and architecture it asks for. The instance runs that one job in `ghcr.io/zeroae/ci-runner:latest`
(GitHub's runner image on Ubuntu 24.04, plus general tools) and then
terminates. The instance has no IAM role, so a job finds no AWS credentials
on it. Short jobs stay on GitHub-hosted runners.
terminates. Each job labelled `ci-fargate-spot` or `ci-fargate` gets a fresh
Fargate task instead, from the same image, which runs that one job and exits.
Neither has an IAM role, so a job finds no AWS credentials on it. Short jobs
stay on GitHub-hosted runners, or go to a Fargate capacity if they need no
Docker.

| Label in `runs-on` | Capacity | For |
|---|---|---|
| `ci-spot` | EC2 Spot, `.xlarge` (`m7i`/`m6i`/`m6a`/`m7a`) | long jobs: a job that runs well over 2 minutes, which is about how long a CI runner takes to come up |
| `ci-ondemand` | EC2 On-Demand, `.xlarge` | critical jobs: a job that, cut off half-way, leaves state a person must repair (`terraform apply`) |
| `ci-spot` | EC2 Spot (`m8i`/`m8a`/`m7i`/`m6i`/`m6a`/`m7a` for `x64`, `m7g`/`m6g`/`m8g` for `arm64`) | long jobs: a job that runs well over 2 minutes, which is about how long a CI runner takes to come up |
| `ci-ondemand` | EC2 On-Demand (the same families) | critical jobs: a job that, cut off half-way, leaves state a person must repair (`terraform apply`) |
| `ci-fargate-spot` | Fargate Spot, **no Docker** | Docker-free jobs that should start fast: about 30 s from queued to started |
| `ci-fargate` | Fargate On-Demand, **no Docker** | Docker-free jobs that must not be cut off half-way |
| (none; `ubuntu-latest`) | GitHub-hosted | everything else |

At most 6 Spot and 2 On-Demand CI runners run at once, across all repos. A
job over the cap waits in a queue until one finishes. If a Spot reclaim kills
A **size label** picks the vCPUs, at 4 GB of memory per vCPU, and an
**architecture label** the CPU:

| Size label | vCPUs / memory | Instance size |
|---|---|---|
| `ci-cpu-2` | 2 / 8 GB | `.large` |
| `ci-cpu-4` | 4 / 16 GB | `.xlarge` |
| `ci-cpu-8` | 8 / 32 GB | `.2xlarge` |
| `ci-cpu-16` | 16 / 64 GB | `.4xlarge` |

On a Fargate capacity the size label sets the task's vCPUs and memory the
same way.

The architecture label is `x64` (Intel and AMD) or `arm64` (Graviton). Both
architectures run the same image, built natively for each. Every job names
one size label and one architecture label: neither has a default, so a job
without them waits for a runner that never comes. A job that used to name only
its capacity got 4 vCPUs and `x64`; add `ci-cpu-4` and `x64` to keep that shape.

The CI runners hold at most 24 Spot, 8 On-Demand, 48 Fargate Spot and 8
Fargate On-Demand vCPUs at once, across all repos (six, two, twelve and two
`ci-cpu-4` runners). A job that would go over that waits in
a queue until enough finish. If an EC2 Spot reclaim kills
a job, a failed check run named `ci-runner / <job name>` appears on its
commit. Nothing retries it automatically: a person re-runs it.

Expand All @@ -35,29 +62,44 @@ it's in both.
```yaml
jobs:
tests:
runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON('["self-hosted","ci-spot"]') || 'ubuntu-latest' }}
runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON('["self-hosted","ci-spot","ci-cpu-8","x64"]') || 'ubuntu-latest' }}
build-arm:
runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON('["self-hosted","ci-spot","ci-cpu-4","arm64"]') || 'ubuntu-latest' }}
apply:
runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON('["self-hosted","ci-ondemand"]') || 'ubuntu-latest' }}
runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON('["self-hosted","ci-ondemand","ci-cpu-2","x64"]') || 'ubuntu-latest' }}
lint:
runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON('["self-hosted","ci-fargate-spot","ci-cpu-2","arm64"]') || 'ubuntu-latest' }}
docs:
runs-on: ${{ vars.ZAE_CI_RUNNER && fromJSON('["self-hosted","ci-fargate","ci-cpu-4","x64"]') || 'ubuntu-latest' }}
```

Each `runs-on` names the full shape: capacity, size and architecture.
A job on `ci-fargate-spot` or `ci-fargate` promises not to use Docker
(no `services:`, `container:`, Docker actions or `docker` commands): a
Fargate task has no Docker daemon
([ADR-0001](docs/adr/0001-fargate-capacities-have-no-docker.md)).

While the variable is unset (the default), every job runs on `ubuntu-latest`,
so the change can merge before the repo is switched on. Once the variable
holds any value, the moved jobs run on our runners. To roll back, unset it.
3. Make the moved job's first step the repo's own toolset check: a script
that names any missing tool and prints each tool's version.
4. Give each moved job a `timeout-minutes` under 120 (110 leaves room). The
sweeper terminates any instance 2 hours after launch, so a job without one
sweeper ends any instance or task 2 hours after launch, so a job without one
dies with its machine instead of failing on a timeout that names it.
5. Move one job first, as a canary (usually the test job). Its queued-to-started
time is the boot time, and that tells you which other jobs are long enough to
move. The runbook's "Measured numbers" has what the canary repo measured.
6. Set the variable (`gh variable set ZAE_CI_RUNNER --body 1`) and watch the
canary.

**The label contract:** `runs-on` must include `self-hosted` and exactly one of
`ci-spot` or `ci-ondemand`. The only other labels allowed are `linux` and `x64`.
A job with any other label is ignored, and it waits for a runner that never
comes.
**The label contract:** `runs-on` must include `self-hosted`, exactly one of
`ci-spot`, `ci-ondemand`, `ci-fargate-spot` or `ci-fargate`, exactly one size label (`ci-cpu-2`, `ci-cpu-4`,
`ci-cpu-8`, `ci-cpu-16`) and exactly one of `x64` or `arm64`. It may also
include `linux`. None of them has a default. A job missing a size or an
architecture label, or with any other label (`aarch64`, an unknown
`ci-cpu-*`, two size labels, both `x64` and `arm64`), is ignored, and it waits
for a runner that never comes.

**What the job gets:** Ubuntu 24.04, the runner as user `runner` (passwordless
sudo), Docker through the host's socket, the `docker compose` and `buildx`
Expand All @@ -73,6 +115,26 @@ a bind mount from any of them works as it does on a GitHub-hosted runner,
`:latest`, and the image is rebuilt weekly, for linux/amd64 and linux/arm64,
each built and smoke-tested on its own native runner.

**On a Fargate capacity** the job gets the same image, user, tools, paths and
50 GB, with three differences:

- **No Docker.** There is no daemon and no socket (the `docker` CLI is still
there, and fails). Picking `ci-fargate-spot` or `ci-fargate` is the job's
promise never to use Docker: no `services:`, no `container:`, no Docker
actions, no `docker build`. A job that breaks it fails at that step
([ADR-0001](docs/adr/0001-fargate-capacities-have-no-docker.md)).
- **Older CPUs.** A Fargate task starts in about 30 s against EC2's 100 s,
but its CPUs are older: a 4-minute Rust build ran about 45% longer. So
Fargate wins for short jobs, and is even at about 4 minutes; for builds
much longer than that, EC2 is faster (the runbook's "Measured numbers").
- **`nproc` over-reports.** Inside a task, `nproc` can show twice the task's
vCPUs. The runner caps cargo's jobs at the task's vCPUs (in
`~/.cargo/config.toml`, so `Swatinem/rust-cache` shares its cache with EC2)
and sets `MAKEFLAGS=-j<n>` and `CI_RUNNER_VCPUS=<n>`. A tool that sizes
itself by `nproc` some other way should be given `-j$CI_RUNNER_VCPUS`.

`/tmp`, `$HOME` and `_work` are the container's own paths on its 50 GB disk.

**Deploy jobs** keep their AWS access as before (typically GitHub OIDC into the
repo's own role). A trust policy that pins `runner_environment: github-hosted`
must drop that condition before a deploy job moves.
Expand Down
5 changes: 5 additions & 0 deletions docs/adr/0001-fargate-capacities-have-no-docker.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
# Fargate capacities have no Docker daemon

The two Fargate capacities, `ci-fargate` and `ci-fargate-spot`, run a job with no Docker daemon and no socket. Picking one is the job author's promise that the job never uses Docker: no `services:`, no `container:` jobs, no Docker actions, no `docker build`. A job that breaks the promise fails at that step, and nothing detects it beforehand.

We accepted that because the jobs it serves (Rust builds, for example) need CPU, memory and disk, not containers. Fargate allows no privileged mode, so the only ways to give a job Docker there are rootless daemons or remote builders, which each add a moving part to every job. Jobs that need Docker stay on the EC2 capacities, `ci-spot` and `ci-ondemand`, which keep the host's daemon.
7 changes: 7 additions & 0 deletions docs/adr/0002-push-the-image-to-ecr-from-ci.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Push the runner image to ECR from CI, not through a pull-through cache

Fargate pulls the whole image (about 820 MB) for every task and caches nothing, so start time depends on lazy loading (SOCI), and that works only from private ECR. The public `image.yml` job therefore pushes the image to an ECR repo that this stack owns, both architectures with a SOCI v2 index, through GitHub OIDC. The OIDC setup follows foresight's `github-deploy-zae.json`: the provider is created only when the account doesn't already have one. EC2 runners keep pulling from GHCR, because an instance with no instance profile can't pull from private ECR.

## Considered Options

- **ECR pull-through cache from ghcr.io.** CI wouldn't change at all. We rejected it for three reasons: ghcr.io upstream needs a GitHub token in Secrets Manager even for a public image, so someone must keep rotating it; a cached `:latest` can be up to 24 hours stale; and no AWS document says SOCI works through the cache, while a cached repo can't be pushed to, so we couldn't add an index ourselves.
7 changes: 7 additions & 0 deletions docs/adr/0003-tolerate-swaps-instead-of-run-labels.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Tolerate swaps instead of requiring a run label

GitHub gives a queued job to any idle runner whose labels cover the job's, so a CI runner often runs a job other than the one it was launched for: a swap. We don't prevent swaps. The label contract requires a capacity, a size and an architecture label, so a swap only ever happens between runners of the same shape. The system repairs the bookkeeping instead. When the webhook gets `workflow_job` `in_progress`, it reads `runner_name`. If that runner was launched for another job, the webhook re-queues that other job, so it isn't stranded, and re-tags the runner's instance or task with the job it actually runs, so a Spot reclaim is reported on the right commit.

## Considered Options

- **A run label and a per-job label, as CodeBuild-hosted runners do** (`codebuild-<project>-${{ github.run_id }}-${{ github.run_attempt }}` plus a unique label per job). We rejected it: every consumer's `runs-on` grows two expressions, matrix jobs need a hand-written label each, and the system would still need to repair the swaps that get through.
Loading
Loading