Skip to content

Ride out crates.io blips in coverage's cargo e2e - #1654

Open
Mikola Lysenko (mikolalysenko) wants to merge 1 commit into
mainfrom
ci-janitor/cargo-net-retry-on-ci
Open

Mikola Lysenko (mikolalysenko) wants to merge 1 commit into
mainfrom
ci-janitor/cargo-net-retry-on-ci

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The coverage job (and with it ci-ok, a required check that also gates the merge queue) failed on PR #1652 at 4f7319c, job 119029890773, for a reason unrelated to the PR:

e2e_redirect_cargo_build (main): ... panicked at crates/socket-patch-cli/tests/e2e_redirect_cargo_build.rs:847:9:
(4b) the reverted checkout fetches from crates.io:
warning: spurious network error (3 tries remaining): [7] Could not connect to server (Failed to connect to static.crates.io port 443 after 5001 ms)
warning: spurious network error (2 tries remaining): ...
warning: spurious network error (1 try remaining): ...
error: failed to download from `https://static.crates.io/crates/cfg-if/1.0.5/download`

This is the same crates.io blip class #1459 fixed for the required cargo … lock-v* legs. Over the last ~200 CI runs (since 2026-10-12) it is the only infrastructure failure of coverage; the other coverage/ci-ok reds were real PR failures. In the merge queue one such blip evicts the entry and costs a full rebuild.

Root cause

#1459 widened cargo's net.retry only when SOCKET_PATCH_CARGO_E2E_REQUIRED is set. The coverage job runs the same real-cargo suites (e2e_redirect_cargo_build, e2e_redirect_cargo_shapes, e2e_vendor_cargo_build, mode_migration_cargo, e2e_safety_cargo_build) as optional, so their cargo children keep the default net.retry=3 (~11s). Optional mode soft-skips only the baseline fixture build; later fetches such as the (4b) revert's cargo fetch --locked are hard assertions, so a blip after the baseline succeeded fails the job.

Fix

cargo_e2e_matrix::cargo_command sets CARGO_NET_RETRY=10 (~80s of cargo's own backoff) when CI is set too, not only on required legs. GitHub Actions always sets CI=true. Optional local runs still keep the default, so an offline machine still skips quickly. An ambient CARGO_NET_RETRY still wins. The old-cargo 127.0.0.1:9 proxy check builds its own Command with CARGO_NET_RETRY=0, so it is unaffected. Only errors cargo classifies as spurious are retried. Checksum, resolver and --locked failures still fail at once.

Proof

  • With a logging cargo shim on PATH, cargo test -p socket-patch-cli --test e2e_redirect_cargo_build:
    • CI unset: all 13 cargo children (build ×4, fetch ×5, run, --version ×3) see CARGO_NET_RETRY unset, as before.
    • CI=true: all 13 see CARGO_NET_RETRY=10.
    • 23 passed in both runs.
  • Ride out crates.io blips in required cargo e2e legs #1459 measured the retry window for this exact error ([7] Could not connect ... after 0 ms): retry=3 fails after 12s, retry=10 rides out a simulated 20s blip and succeeds after 21s.
  • rustfmt --check is clean on the touched file. cargo clippy -D warnings reports nothing in it. The pre-existing errors in tests/common/mod.rs and tests/prebuilt_common/mod.rs are unchanged.

No tests removed or moved.

https://claude.ai/code/session_017UzN47eRAtVax9Sbom2ffx


Generated by Claude Code

The coverage job runs the real-cargo e2e suites without
SOCKET_PATCH_CARGO_E2E_REQUIRED, so their cargo children kept cargo's
default net.retry=3 (~11s). Their baseline fetch soft-skips when
crates.io is down, but later fetches are hard assertions: on PR #1652
a static.crates.io connect blip during the (4b) revert's
`cargo fetch --locked` in e2e_redirect_cargo_build failed coverage and
with it ci-ok, which also gates the merge queue.

cargo_command now sets CARGO_NET_RETRY=10 (~80s of cargo's own
backoff, as #1459 does for required legs) when CI is set as well.
Optional local runs still keep the default so an offline machine skips
fast, and an ambient CARGO_NET_RETRY still wins.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017UzN47eRAtVax9Sbom2ffx
@mikolalysenko Mikola Lysenko (mikolalysenko) added the ci-janitor Opened by the CI janitor routine (flakes, redundant tests, CI perf) label Oct 21, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit afb77c3. Configure here.

@mikolalysenko Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Oct 21, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Burn-down agent: labeled Ready for review.

  • Head: afb77c3
  • CI: 35/35 checks green (19 skipped)
  • Bugbot: reviewed this head (check Cursor Bugbot success), no findings, no open review threads
  • Mergeable, no conflicts; no CHANGELOG.md change

Slack announcement not sent: this run has no Slack send tool.


Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Burn-down agent: labeled Ready for review at afb77c3.

  • CI: 35/35 checks green on the head (19 skipped), no main-wide failures.
  • Bugbot: reviewed afb77c3, no findings; no open review threads.
  • Mergeable against main, no CHANGELOG changes.
  • Slack announcement not sent this run (no Slack send tool available); the next run will retry.

Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Final review brief (for the approver)

What it does. The real-cargo e2e suites now widen cargo's net.retry to 10 (~80s of cargo's own backoff) whenever CI is set, not only on the required lock-v* legs. The coverage job runs those suites as optional, so a crates.io blip after the baseline fetch failed ci-ok (seen on #1652). Test-harness only; no product code changes.

Risk: low. One condition in a test helper. Effects: on CI a genuinely offline runner takes up to ~80s longer per cargo child before failing; an explicit CARGO_NET_RETRY still wins; CI=0/CI=false are treated as not-CI. Cargo only retries errors it classifies as spurious, so checksum/--locked failures still fail immediately.

Look here

  • cargo_e2e_matrix/mod.rs:95-100:`` on_ci()
  • cargo_e2e_matrix/mod.rs:115: the widened condition
  • cargo_e2e_matrix/mod.rs:83-94:``` rationale comment

Verified. Read the full diff and the helper it changes; no test skipped, removed or weakened, no workflow or required-check change. CI 54/54 settled on afb77c3 (35 success, 19 path-skipped), ci-ok and clippy green. Bugbot reviewed afb77c3: no findings; no review threads. Mergeable, CHANGELOG.md untouched.

Changes I made. None.

Open questions. None.

Auto-merge is armed, so approving sends it straight to the merge queue.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-janitor Opened by the CI janitor routine (flakes, redundant tests, CI perf) Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants