Skip to content

Notifier: name the force-cancel for a lost job GitHub keeps in progress - #50

Merged
sodre merged 2 commits into
mainfrom
reclaim-force-cancel-hint
Nov 16, 2026
Merged

sodre merged 2 commits into
mainfrom
reclaim-force-cancel-hint

Conversation

@sodre

@sodre sodre commented Nov 16, 2026

Copy link
Copy Markdown
Member

Closes #48

Summary

A Spot reclaim can leave the lost job in_progress after its machine is gone. While it is stuck, gh run rerun is refused and gh run cancel ends nothing. The reclaim check run's summary now names the step that works. Its title is unchanged, because the notifier finds its reports by that title.

 ci-runner / <job>   failure   "Spot capacity reclaimed: re-run this job"
-AWS reclaimed the Spot instance (i-…) running `tests`. Re-run the job: <run>
+AWS reclaimed the Spot instance (i-…) running `tests`. If the job failed, re-run it: <run>
+
+If the job still shows in progress, the run cannot be re-run, and a plain cancel does not end it.
+Force-cancel the run, then re-run its failed jobs:
+`gh api -X POST repos/<repo>/actions/runs/<run id>/force-cancel`.
+A force-cancel ends every job still running in the run: let the others finish first.

Docs:

  • docs/runbook.md: "Read a lost job" covers the stuck case.
  • docs/traps.md: a new row for the symptom.
  • docs/stack.md: step 6 matches the new summary.

Evidence

  • Before: a reclaim report tells how to unstick a lost job GitHub keeps in progress fails. The summary has no gh api -X POST repos/zeroae/app/actions/runs/444/force-cancel.
    After: it passes. make test gives 183 pass, 0 fail. make build gives 78001 of 1048576 bytes.
  • The incident: GitHub's API shows job 133599865684 stayed in_progress from 20:46:23 until it completed as cancelled at 21:01:02, the moment of the force-cancel.

Merge Danger

Door: two-way

Only the text of new reclaim reports changes. Reports already on commits keep their old summary, and the title that skipping and clearing depend on is the same.

Blast Radius: wording

It takes effect after the next make deploy.

🤖 Generated with Claude Code

sodre and others added 2 commits November 16, 2026 16:18
GitHub may keep a job in_progress after its Spot machine is gone: a re-run
is refused and a plain cancel ends nothing, so "Re-run the job" alone can't
be followed. The reclaim report's summary now gives the run's force-cancel,
and warns that it ends the run's other running jobs too. The title is
unchanged, since the notifier finds its reports by it. The runbook's "Read a
lost job" and the traps table describe the stuck case.

Closes #48

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…raps match it

Review of #48: the summary's first instruction now reads "If the job failed,
re-run it", so a reader in the stuck case doesn't meet the failing step first.
stack.md's step 6 names the re-run and the force-cancel's warning; the traps
row uses zeroae/<repo> as the runbook does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sodre
sodre merged commit 6340c67 into main Nov 16, 2026
2 checks passed
@sodre
sodre deleted the reclaim-force-cancel-hint branch November 16, 2026 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A job whose Spot instance is reclaimed stays in progress until its run is force-cancelled

1 participant