Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion GLOSSARY.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,4 +40,4 @@ The most vCPUs that the CI runners of one capacity may hold at once. A job that
_Avoid_: "cap" in prose (that was the old runner count). The parameters that set the budgets are still named `*VcpuCap` (`SpotVcpuCap`, `OnDemandVcpuCap`, `FargateSpotVcpuCap`, `FargateVcpuCap`).

**Spot reclaim**:
AWS taking back Spot capacity, EC2 or Fargate, while a job runs. The job is lost and is reported as one on its commit.
AWS taking back Spot capacity, EC2 or Fargate, while a job runs. The job is lost, unless it finishes inside AWS's 2-minute warning, and a lost job is reported as a reclaim on its commit.
15 changes: 13 additions & 2 deletions docs/runbook.md
Original file line number Diff line number Diff line change
Expand Up @@ -295,6 +295,15 @@ Its summary names the instance (or the task's id) and links the run. That is
the whole story: re-run the failed job. Nothing retries it automatically.
One check run per job: the notifier skips a job whose commit has it already.

AWS warns 2 minutes before it takes the machine, and a job can finish inside
that. When the job ends with `success`, the check run turns `neutral`, titled
`The job finished despite the Spot reclaim warning`: no re-run is needed (the
webhook tells the notifier; see `docs/stack.md`, "Reporting a Spot reclaim").
A `failure` check run whose job passed anyway is safe to ignore. A re-run that
passes turns the first attempt's report `neutral` too. The notifier reports
only a job the machine's CI runner was running: a CI runner that took no job
(another one served its launch job) is not reported.

To list those check runs on a commit:

```bash
Expand All @@ -304,8 +313,10 @@ gh api repos/zeroae/<repo>/commits/<sha>/check-runs \

A job lost **without** that check run wasn't a Spot reclaim. Look in the
notifier's log for a line `reported <instance or task id>: <repo> job <id>`
(or `already reported ...`). An instance from `make hand-launch`, or a task
run by hand, carries no job tags, so the notifier ignores it and logs
(or `already reported ...`, or `survived: ...`). A line
`ignored <id>: its runner is not running job <id> (...)` means the machine's
CI runner ran no job when the warning came. An instance from `make hand-launch`,
or a task run by hand, carries no job tags, so the notifier ignores it and logs
`ignored: <instance> is no job instance of this stack` (`... no job task
...` for a task). A task's stop code is in `describe-tasks` (below):
`SpotInterruption` is a reclaim. A job that sits
Expand Down
65 changes: 55 additions & 10 deletions docs/stack.md
Original file line number Diff line number Diff line change
Expand Up @@ -209,16 +209,28 @@ App's webhook points at), for each delivery:
3. **Fence 3:** refuses (`403`, logged) unless `repository.visibility` is
`private` and the owner is `GITHUB_ORG`.
4. On `in_progress`, repairs a swap (below) and answers.
5. Ignores (`200`) actions other than `queued`, and jobs outside the label
5. On `completed`, tells the notifier when the job succeeded on a runner of
this stack (below) and answers.
6. Ignores (`200`) actions other than `queued`, and jobs outside the label
contract (case-insensitive): `self-hosted`, exactly one of
`ci-spot`/`ci-ondemand`/`ci-fargate-spot`/`ci-fargate`, exactly one size label (`ci-cpu-2`, `-4`, `-8`,
`-16`), exactly one of `x64`/`arm64`, and optionally `linux`. None has a
default: a missing size or architecture label, an unknown `ci-cpu-*`, or
both `x64` and `arm64`, are outside it.
6. Enqueues `{repo, job_id, run_id, head_sha, job_name, label, vcpus, arch,
7. Enqueues `{repo, job_id, run_id, head_sha, job_name, label, vcpus, arch,
labels, installation_id}` (`label` is the capacity label, lower case;
`vcpus` is 2, 4, 8 or 16; `arch` is `x64` or `arm64`) and answers `202`.

**Telling the notifier a job succeeded.** On `workflow_job` `completed` with
conclusion `success`, and a `runner_name` that is `${stack}-<14-digit
stamp>-<job id>` (a job on a GitHub-hosted runner, another stack's, or none is
ignored, `200`), the webhook invokes the notifier asynchronously
(`lambda:InvokeFunction` on it, env `NOTIFIER_FUNCTION`) with `{jobSucceeded:
{repo, job_id, run_id, head_sha, job_name, installation_id}}` and answers
`202`. The App's `workflow_job` subscription already delivers every action
(about 3 deliveries per job: `queued`, `in_progress`, `completed`), so nothing
about the App changes. The webhook does no GitHub call itself.

**Swap repair** (ADR-0003). GitHub gives a queued job to any idle runner whose
labels cover it, so a runner launched for job Y may take job X. Left alone,
X's own message is later dropped as not queued, and Y never gets a runner.
Expand Down Expand Up @@ -254,7 +266,7 @@ tags and repairs again. Re-queueing a job that another runner already took is
harmless: the launcher drops it.

Its role allows `GetSecretValue` on the secret, `SendMessage` on the queue,
`ec2:DescribeInstances`, and `ec2:CreateTags` only on instances tagged
`lambda:InvokeFunction` on the notifier, `ec2:DescribeInstances`, and `ec2:CreateTags` only on instances tagged
`ci-runner:stack=<stack>` and only for the five job tag keys. On ECS:
`ecs:ListTasks` with the condition `ecs:cluster` = the cluster, and
`ecs:DescribeTasks` and `ecs:TagResource` only on the cluster's tasks, the
Expand Down Expand Up @@ -470,18 +482,51 @@ For each event it:
as is a task whose tags are gone.
2. Signs an App JWT, finds the repo's installation
(`GET /repos/{repo}/installation`) and gets its token.
3. For a task, reads the job's real name from
`GET /repos/{repo}/actions/jobs/{job_id}` (the tag has `_` for brackets and
commas), falling back to the tag.
4. Skips the job if the commit already has its reclaim check run (same name,
`external_id` the job id, the title below), so a task's later changes and a
repeated event report nothing more.
5. Creates a check run on the head sha: name `ci-runner / <job name>` (or
3. Reads the job from GitHub (`GET /repos/{repo}/actions/jobs/{job_id}`), the
job the tags name: the one the machine was launched for, or the one it took
in a swap (the webhook re-tags it). It **reports only a job the machine's CI
runner is running**: `status` `in_progress` and `runner_name` equal to the
machine's `ci-runner:runner` tag. Otherwise it logs `ignored <id>: its runner
is not running job <id> (...)` and posts nothing: the job is `queued`, or
`completed`, or another CI runner runs it. The usual case is an idle CI
runner: another CI runner took the job it was launched for, it took none,
and so its tags are still the launch tags. A swapped job that the webhook
has not yet re-tagged, in the moments after it started, is missed the same
way.
4. Takes the job's real name from that job, falling back to the tag: a tag is
cut at 256 characters, and a task's has `_` for brackets and commas. The
webhook's word (step 8) carries GitHub's name too, so the two meet.
5. Skips the job if the commit already has its reclaim check run (same name,
`external_id` the job id, the title below or the survived title), so a task's
later changes and a repeated event report nothing more.
6. Creates a check run on the head sha: name `ci-runner / <job name>` (or
`job <id>` with no name), `completed`, conclusion `failure`, title
`Spot capacity reclaimed: re-run this job`, a summary naming the job and
the instance or task and linking the run, and `details_url` the job's page.
An EC2 reclaim and a Fargate one give the same check run, but for the
summary's machine.
7. Reads the job once more after posting: a job that finished with `success`
while this ran is cleared as in step 8.
8. **Clears a report the job survived.** AWS warns 2 minutes before it takes
the machine, and a job can finish inside that. A job that finishes with
`success` was not lost, so its reclaim check run changes to `neutral`,
titled `The job finished despite the Spot reclaim warning`, and the notifier
logs `survived: <repo> <job name>`. Three paths clear a report:
- **the webhook's word** (above): the notifier gets the token (the event
carries the installation id), lists the commit's check runs named `ci-runner
/ <job name>`, and clears each reclaim `failure` whose `details_url` is
under the job's run. That takes in the report of an earlier attempt of the
job: a re-run's job has a new id, but the same run. With none to clear it
logs `nothing to clear` and changes nothing;
- **a retry or a later change** of a reclaim event that finds its own
standing `failure` (step 5) for a job that has since finished with
`success`;
- **step 7**, for the word that came before the report existed.

A job that ends with any other conclusion keeps the `failure`. If the
webhook's invoke of the notifier fails, the delivery fails (`502`): redeliver
it from the App's advanced settings, or the retry and later-change paths
clear the report.

A GitHub failure fails the invocation, and EventBridge's async invoke retries
it. Two events handled at the same moment can still report twice.
Expand Down
95 changes: 78 additions & 17 deletions functions/notifier.js
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
// Fargate Spot task's stop (ECS Task State Change, stopCode SpotInterruption).
// If the instance or task is this stack's and carries a job's launch tags (see
// launcher.js), it creates a failed check run on the job's commit through the App.
// The webhook also invokes it when a job of this stack's runners succeeds: a
// reclaim report for that job turns neutral.
// The App JWT is a copy of the launcher's: bundling a shared module would cost
// more template bytes than the copy does.
const crypto = require('node:crypto');
Expand All @@ -20,6 +22,9 @@ function appJwt(appId, pem, nowMs) {
// keyed without the ci-runner: prefix.
const tagsOf = (tags) => Object.fromEntries((tags || []).map((x) => [(x.Key ?? x.key).replace(/^ci-runner:/, ''), x.Value ?? x.value]));

const RECLAIMED_TITLE = 'Spot capacity reclaimed: re-run this job';
const SURVIVED_TITLE = 'The job finished despite the Spot reclaim warning';

function createHandler({ secret, fetch, ec2, ecs, now, env, log = console.log }) {
async function gh(method, path, token, body) {
const res = await fetch(API + path, {
Expand All @@ -31,7 +36,52 @@ function createHandler({ secret, fetch, ec2, ecs, now, env, log = console.log })
return json;
}

// An installation token for the repo. The webhook passes the installation id
// it already has; an EventBridge event carries none.
async function tokenFor(repo, installationId) {
const s = await secret.get();
if (!s.private_key) throw new Error('the App is not set up');
const jwt = appJwt(s.app_id, s.private_key, now());
const installation = installationId || (await gh('GET', `/repos/${repo}/installation`, jwt)).id;
return (await gh('POST', `/app/installations/${installation}/access_tokens`, jwt)).token;
}

// The commit's reclaim check runs under a job's name. A task's reclaim
// matches on each of its changes, and EventBridge may repeat or retry an
// event: one check run per job is enough.
async function reportsNamed(repo, token, sha, name) {
const { check_runs: existing } = await gh('GET', `/repos/${repo}/commits/${sha}/check-runs?filter=all&check_name=${encodeURIComponent(name)}`, token);
return existing.filter((c) => c.output && [RECLAIMED_TITLE, SURVIVED_TITLE].includes(c.output.title));
}

// A job that finished with success was not lost: its check run no longer
// asks for a re-run.
async function clear(repo, token, checkId, job, run) {
await gh('PATCH', `/repos/${repo}/check-runs/${checkId}`, token, {
conclusion: 'neutral',
output: { title: SURVIVED_TITLE, summary: `AWS warned that it would reclaim the Spot machine running \`${job}\`, but the job finished with success. No re-run is needed: ${run}` },
});
log(`survived: ${repo} ${job}`);
}

// The webhook says a job of this stack's runners finished with success. A
// job that had a reclaim report (AWS warns 2 minutes before it takes the
// machine, and a job can finish inside that) is cleared. So is the report
// of an earlier attempt of the job, which a re-run of the run left behind:
// its details_url is under the same run.
async function jobSucceeded(j) {
// The same fence as a reclaim event's: only the org's repos.
if (String(j.repo).split('/')[0] !== env.GITHUB_ORG) return log(`ignored: ${j.repo} is outside ${env.GITHUB_ORG}`);
const token = await tokenFor(j.repo, j.installation_id);
const run = `https://github.com/${j.repo}/actions/runs/${j.run_id}`;
const standing = (await reportsNamed(j.repo, token, j.head_sha, `ci-runner / ${j.job_name}`))
.filter((c) => c.output.title === RECLAIMED_TITLE && (c.details_url || '').startsWith(`${run}/`));
if (!standing.length) return log(`nothing to clear: ${j.repo} job ${j.job_id}`);
for (const c of standing) await clear(j.repo, token, c.id, j.job_name, run);
}

return async (event) => {
if (event.jobSucceeded) return jobSucceeded(event.jobSucceeded);
const d = event.detail || {};
let id, machine, t;
if (d.taskArn) {
Expand All @@ -52,33 +102,44 @@ function createHandler({ secret, fetch, ec2, ecs, now, env, log = console.log })
return log(`ignored: ${id} is no job ${d.taskArn ? 'task' : 'instance'} of this stack`);
}

const s = await secret.get();
if (!s.private_key) throw new Error('the App is not set up');
const jwt = appJwt(s.app_id, s.private_key, now());
const { id: installation } = await gh('GET', `/repos/${repo}/installation`, jwt);
const { token } = await gh('POST', `/app/installations/${installation}/access_tokens`, jwt);

// A task's job-name tag has `_` for what ECS refuses (brackets, commas):
// the check run carries the real name, from GitHub.
const ghJob = d.taskArn && (await gh('GET', `/repos/${repo}/actions/jobs/${jobId}`, token).catch(() => null));
const job = (ghJob && ghJob.name) || t['job-name'] || `job ${jobId}`;
const token = await tokenFor(repo);
const jobPath = `/repos/${repo}/actions/jobs/${jobId}`;
const ghJob = await gh('GET', jobPath, token);
// The check run carries the job's real name, from GitHub, as the webhook's
// word does: a task's job-name tag has `_` for what ECS refuses (brackets,
// commas), and every tag is cut at 256 characters.
const job = ghJob.name || t['job-name'] || `job ${jobId}`;
const run = `https://github.com/${repo}/actions/runs/${t['run-id']}`;
const name = `ci-runner / ${job}`;
const title = 'Spot capacity reclaimed: re-run this job';
// A task's reclaim matches on each of its changes, and EventBridge may
// repeat an event: one check run per job is enough.
const { check_runs: existing } = await gh('GET', `/repos/${repo}/commits/${t['head-sha']}/check-runs?filter=all&check_name=${encodeURIComponent(name)}`, token);
if (existing.some((c) => c.external_id === jobId && c.output && c.output.title === title)) return log(`already reported ${id}: ${repo} job ${jobId}`);
await gh('POST', `/repos/${repo}/check-runs`, token, {
const finishedWithSuccess = (j) => j.status === 'completed' && j.conclusion === 'success';

const reported = (await reportsNamed(repo, token, t['head-sha'], name)).find((c) => c.external_id === jobId);
if (reported) {
// An earlier event's report, left standing: the job finished with
// success after the webhook's word, or before the report existed.
if (reported.output.title === RECLAIMED_TITLE && finishedWithSuccess(ghJob)) return clear(repo, token, reported.id, job, run);
return log(`already reported ${id}: ${repo} job ${jobId}`);
}
// Only a job this machine's CI runner is running can be lost with it. The
// tags name the job the machine was launched for, or took in a swap (the
// webhook re-tags it); a CI runner that took no job (another one served
// its launch job) is idle, and a completed job is no loss.
if (ghJob.status !== 'in_progress' || ghJob.runner_name !== t.runner) {
return log(`ignored ${id}: its runner is not running job ${jobId} (${ghJob.status}, runner ${ghJob.runner_name || 'none'})`);
}
const { id: checkId } = await gh('POST', `/repos/${repo}/check-runs`, token, {
name, head_sha: t['head-sha'], external_id: jobId,
status: 'completed', conclusion: 'failure', completed_at: new Date(now()).toISOString(),
details_url: `${run}/job/${jobId}`,
output: {
title,
title: RECLAIMED_TITLE,
summary: `AWS reclaimed the Spot ${machine} running \`${job}\`. Re-run the job: ${run}`,
},
});
log(`reported ${id}: ${repo} job ${jobId}`);
// The job may have finished while this ran, and the webhook's word then
// found no report to clear.
if (finishedWithSuccess(await gh('GET', jobPath, token))) await clear(repo, token, checkId, job, run);
};
}

Expand Down
Loading
Loading