Skip to content

fix(sandbox): report socket-blocked isolation probes - #511

Merged
datacas merged 2 commits into
mainfrom
fix/508-no-network-sandbox-suite
Nov 17, 2026
Merged

datacas merged 2 commits into
mainfrom
fix/508-no-network-sandbox-suite

Conversation

@datacas

@datacas datacas commented Nov 17, 2026

Copy link
Copy Markdown
Owner

Summary

Addresses #508 by converting an OSError while creating, binding, or starting the isolation probe listener into an explicit unavailable (not_verified) result. The existing dispatch evidence path records the cause. The HeadMovedTests fixture now uses a temporary repository with a local bare origin, so its ref audit needs no network or real repository remote.

Diagnosis

Issue #508 records the no-network Codex reproduction on PR #502: 14 failures and 8 errors, including the uncaught socket denial in preflight and installed-cycle dispatches, plus a real-origin failure in HeadMovedTests. Source inspection confirmed that socket setup occurred before the probe's error handler and that the test ran with cwd=str(ROOT). Before editing, I reran the affected prerequisite and CI-policy modules in this stage; they passed because this executor allowed those socket and network operations, so those runs did not independently reproduce the sandbox-specific denial.

The invariant is that inability to verify isolation is recorded with its operating-system cause and never crashes a dispatch or becomes a pass. The test fixture must also keep ref auditing local. The fix catches the listener's OSError and returns False with the original cause; the dispatch already records this result as preparation_isolation evidence and marks the preflight unverified. The head-movement test uses a disposable local bare remote.

Related work was searched by the affected symbols and sandbox terms: #274 and #288 are closed and cover different Claude sandbox and prerequisite failures; #507 is an open Docker-state issue with a different cause. No other open item is addressed here.

Work units

Unit Behaviour Files Tests Verification Rollback
WU-1 Report socket-denied isolation as unverified evidence and make head-movement ref auditing hermetic scripts/native_prerequisites.py, tests/test_native_prerequisites.py, tests/test_ci_policy.py, docs/dependency-preparation.md, docs/capability-parity-audit.md Prerequisite, installed-cycle, and CI-policy suites Configured suite, package validator, BasedPyright; Codex no-network context remains unverified below independent

Verification (agent-reported)

  • python3 -m unittest discover -s tests -p test_native_prerequisites.py — passed, 28 tests.
  • python3 -m unittest discover -s tests -p test_installed_cycle.py — passed, 18 tests.
  • python3 -m unittest discover -s tests -p test_ci_policy.py — passed, 54 tests; the head-movement fixture uses a local bare origin.
  • python3 -m unittest discover -s tests — passed, 2,641 tests, 1 skipped, 934 seconds. This run was in the selected stage executor and was not a no-network Codex sandbox run; it preceded a final defensive close-error guard, which was checked with the prerequisite suite and BasedPyright.
  • python3 scripts/validate-package.py — passed.
  • python3 -m unittest — exit 0, 0 tests discovered by this configured command.
  • basedpyright — passed, 0 errors, 0 warnings, 0 notes.

The no-network Codex sandbox context was not available to this Claude -p dispatch, so AC-1 and AC-3 remain unverified for that exact context. The suite was not rerun through a nested Codex sandbox. The mocked socket-denial test verifies the unavailable result and preserves the real error text; the full suite and focused integration suites are reported separately.

Security review ran. It found no confirmed vulnerabilities in the changed surface. The diff adds no credential access, permission widening, command execution, or external data handling. The ref-audit fixture uses only a disposable local bare repository.

@datacas

datacas commented Nov 17, 2026

Copy link
Copy Markdown
Owner Author

Implementation summary

Issue #508 is implemented in commit 787964f955e246677f6b95c569441e1e05312078. The isolation probe now returns an unavailable result with the socket error, and the head-movement test uses a local bare origin.

Verification: 2,641 full-suite tests passed with one skip; focused prerequisite (28), installed-cycle (18), and CI-policy (54) suites passed; package validation and BasedPyright passed. python3 -m unittest exited 0 but discovered 0 tests. The exact no-network Codex sandbox context was not available in this Claude -p dispatch, so AC-1 and AC-3 remain unverified there. Security audit ran: no confirmed vulnerabilities found.

@datacas

datacas commented Nov 17, 2026

Copy link
Copy Markdown
Owner Author

[CCR-20261117-001] · senior_reviewer · anthropic/claude-opus-5-5→? · high · schema:2

Verdict: CHANGES_REQUESTED on 787964f955e246677f6b95c569441e1e05312078. 1 finding, 1 blocking.

The probe fix is correct. It fails closed: a denied listener gives not_verified and is never a pass. HeadMovedTests is now hermetic. The full suite passes here (2641 tests, 1 skipped).

  • REV-001 (medium, blocks): AC-1 is still unmet for 4 of the 5 PreflightTests. When the listener cannot be created they fail or error, and none of them skips with an environment cause. In test_probe_failure_and_timeout_are_not_verified the probe now returns before the runner runs, so runner.call_args.args raises AttributeError. That is the unexplained crash AC-3 rules out. Fix: give the mocked-runner tests a working listener, and skip the real-process tests with the OS error as the cause when no localhost listener can be bound.

Security: the audit ran, no findings. CI: 4 checks passed, 2 Windows shards pending.

Review details, findings, verification and CI

[REV-001] · medium · open · - · blocks:yes — PreflightTests still fail or error when socket creation is denied (AC-1, AC-3)

Path: tests/test_native_prerequisites.py (lines 289, 313, 374, 391); behaviour change in scripts/native_prerequisites.py:284-295.

isolation_probe now returns (False, "<kind> isolation probe unavailable: …") before it builds the script or calls runner. These are the five PreflightTests that reach a real socket.socket(). Under a socket denial, as in the no-network Codex sandbox of #508:

  • test_srt_probe_gets_managed_rg_at_the_process_boundary (L289): assertTrue(ok, evidence) → FAIL.
  • test_selected_system_rg_is_not_shadowed_by_stale_managed_rg (L313): assertTrue(...[0]) → FAIL, with no message.
  • test_probe_failure_and_timeout_are_not_verified (L374): runner.call_args is None, so .args → ERROR AttributeError.
  • test_nonzero_real_process_boundary (L391): assertIn("Creating new namespace failed", evidence) → FAIL.
  • test_lying_probe_success_is_not_verified (L384): passes.

AC-1 requires these tests to "pass or skip with an explicit environment cause". AC-3 forbids unexplained crashes. The new test test_socket_creation_failure_is_reported_as_unverified covers the product path but not this suite behaviour. Suggested fix: patch socket.socket with a working stub in the tests whose runner is mocked, so their assertions do not depend on host sockets. For the real-process tests, add a small helper that tries to bind 127.0.0.1:0 and calls self.skipTest(f"local sockets denied: {error}") when that fails.

Evidence: code inspection of the control flow. I tried to replay the tests under a simulated PermissionError with a helper script in $TMPDIR. The review-stage write policy refused it ([layer=toolkit_hook], rule edit_rule), so this finding is not runtime-verified.

Reviewed scope: the full accumulated diff origin/main...787964f (merge base f66b2ba, 119 lines, 5 files): scripts/native_prerequisites.py, tests/test_native_prerequisites.py, tests/test_ci_policy.py, docs/dependency-preparation.md, docs/capability-parity-audit.md. I also read the call sites scripts/executors.py:1271 (bwrap → static_only) and :1346 (srt → preparation_isolation evidence, status: unverified), the related dispatch tests in tests/test_stage_dispatch.py, and the git_repo fixture.

Correctness notes (no finding): the listener's OSError is caught for create, bind and listen. A partly created socket is closed. TemporaryDirectory still cleans up on the early return. Both callers already treat False as unverified: srt adds the preflight text to the task, bwrap degrades to static_only. HeadMovedTests uses a disposable repository and a local bare origin, so git ls-remote needs no network.

Acceptance criteria

AC State Evidence
AC-1 not met REV-001. InstalledCycleTests were not run in a no-network Codex sandbox (not available to this Claude stage)
AC-2 verified HeadMovedTests ran: 3 tests OK, with the local bare origin
AC-3 not met REV-001 (AttributeError in L374). A no-network Codex run was not available
AC-4 verified (unit) The new probe test, plus the existing preparation_isolation and status: unverified path in executors.py:1346-1349
AC-5 verified docs/dependency-preparation.md and parity audit row PA-46 (real limit)

Verification (executed in this review's sandbox at the PR head)

Check Command Result
configured-test python3 -m unittest discover -s tests pass: 2641 tests, 1 skipped, 499.9 s, exit 0
verify python3 scripts/validate-package.py pass: VALID: 16 skills …
verify python3 -m unittest exit 0, 0 tests discovered
verify basedpyright pass: 0 errors, 0 warnings, 0 notes
focused python3 -m unittest discover -s tests -p test_native_prerequisites.py pass: 28 tests
focused python3 -m unittest discover -s tests -p test_ci_policy.py -k HeadMovedTests pass: 3 tests

Not verified: a run inside a no-network Codex sandbox. The nested Codex sandbox is not available from a Claude stage, and the simulation helper was refused (see REV-001).

Security triage: the project declares no security_review.always_when. The mandatory minimum (CI, secrets, agent config) matched no changed path. My own reading fired the audit, because the change touches security tooling (the sandbox isolation probe), and the runtime plan sets security_required. cc-security-review ran on the full diff. Result: no confirmed vulnerabilities and no potential risks. The change fails closed, because a denied probe is never reported as verified. The error text includes only the OS error string. The test fixture runs git with list argv inside a temporary directory. No credentials, permissions or external inputs are added.

CI (parallel policy, read once): Classify the change pass. Skills, manifests, and Bash installer pass. Windows test shard (windows-1) pass. Windows test shard (windows-2) pass. Windows test shard (windows-3) pending. Windows test shard (windows-4) pending.

Residual risk: AC-1 and AC-3 are unverified in the real no-network Codex context. Other socket users in the suite, such as AF_UNIX binds, may still fail there and were not audited by execution.

@datacas

datacas commented Nov 17, 2026

Copy link
Copy Markdown
Owner Author

Resolution update — PR #511 at 7eebf0e

[CCT-20261117-001] · cheap_coder · openai/gpt-6-luna→? · high · triaged:787964f955e246677f6b95c569441e1e05312078 · schema:2

PARTIALLY_RESOLVED. REV-001 was valid: socket denial could crash mocked-runner tests or fail real-process tests. Mocked tests now stub the listener; real-process tests skip with the OS cause. Focused (29) and full (2,642; 1 skipped) tests pass. The no-network Codex sandbox check remains unverified in this Claude dispatch. Security audit ran; no findings. Remote checks on the pushed head: 2 pending.

[REV-001] · medium · resolved · valid · blocks:yes — PreflightTests still fail or error when socket creation is denied (AC-1, AC-3)

Resolution history, verification and receipts

Finding history

[CCR-20261117-001] · senior_reviewer · anthropic/claude-opus-5-5→? · high · schema:2

[REV-001] · medium · open · - · blocks:yes — PreflightTests still fail or error when socket creation is denied (AC-1, AC-3)

The review found that two tests with mocked runners crashed because listener creation stopped the runner from being called, while three real-process tests depended on localhost socket creation and failed without an explicit environment cause. Disposition assigned against triage head 787964f: valid. The fix adds a localhost bind preflight that skips with the OS error, uses a MagicMock listener for mocked runners, and tests the skip message. No inline review threads were present.

Verification (agent-reported; tests.version 2; head 7eebf0e)

ID Context Result Evidence
configured-test project pass python3 -m unittest discover -s tests: 2,642 tests, 1 skipped, exit 0
ac-1-preflight-tests project pass python3 -m unittest discover -s tests -p test_native_prerequisites.py: 29 tests, exit 0
ac-1-installed-cycle-tests project pass python3 -m unittest discover -s tests -p test_installed_cycle.py: 18 tests, exit 0
ac-2-head-moved-tests project pass python3 -m unittest discover -s tests -p test_ci_policy.py: 54 tests, exit 0
ac-1-3-no-network-codex-suite no-network-codex-sandbox unverified Not run in the requested context. This is a Claude -p dispatch; repository routing classifies nested Codex from Claude as unverified until probed, and this dispatch supplied no host-side Codex sandbox execution capability. The regular full suite and socket-denial regression passed but do not replace this context.
verify:python3 scripts/validate-package.py project pass VALID: 16 skills; exit 0
verify:python3 -m unittest project pass exit 0; 0 tests discovered
verify:basedpyright project pass 0 errors, 0 warnings, 0 notes

Fix-delta review: clean against trusted base 787964f. Security review ran on the fix delta: no confirmed vulnerabilities. Reviewed surface: tests/test_native_prerequisites.py, including socket creation/bind handling in test fixtures. Not reviewed: the full runtime implementation beyond the already reviewed accumulated PR diff. Residual risk: the required no-network Codex suite remains unverified.

Remote checks (read once after push; head 7eebf0e)

  • Classify the change — pending
  • Skills, manifests, and Bash installer — pending
  • Counts: 0 passed, 0 failed, 2 pending

Stage receipts

@datacas

datacas commented Nov 17, 2026

Copy link
Copy Markdown
Owner Author

[CCR-20261117-002] · senior_reviewer · anthropic/claude-opus-5-5→? · high · schema:2

Verdict: REVIEW_PASSED_AWAITING_CI on 7eebf0e018f43524b82a9a32206572d671c26590. No open findings and no new ones.

  • REV-001 resolved. Cause: when socket creation was denied, the tests with a mocked runner crashed and the real-process tests failed. Fix: the mocked-runner tests patch socket.socket with a MagicMock. require_local_listener skips the 3 real-process tests with the OS error as the cause. Test: test_local_listener_failure_skips_with_os_cause runs the skip path under a simulated PermissionError. All 29 preflight tests pass.

Security: the audit ran again, no findings. Tests: the required ac-1-3-no-network-codex-suite was not run (nested Codex not available), so the conclusion is not_verified. CI: 3 passed, 3 Windows shards pending.

Rereview details, finding history, verification and CI

Earlier run lines

[CCR-20261117-001] · senior_reviewer · anthropic/claude-opus-5-5→? · high · schema:2

[CCT-20261117-001] · cheap_coder · openai/gpt-6-luna→? · high · triaged:787964f955e246677f6b95c569441e1e05312078 · schema:2

Findings

[REV-001] · medium · resolved · valid · blocks:yes — PreflightTests still fail or error when socket creation is denied (AC-1, AC-3)

Path: tests/test_native_prerequisites.py.

This is how each of the 5 socket-dependent PreflightTests now behaves when socket creation is denied:

  • test_probe_failure_and_timeout_are_not_verified and test_lying_probe_success_is_not_verified: the listener is a MagicMock, so the runner is always called. Both pass without host sockets.
  • test_srt_probe_gets_managed_rg_at_the_process_boundary, test_selected_system_rg_is_not_shadowed_by_stale_managed_rg and test_nonzero_real_process_boundary: require_local_listener() runs first and calls skipTest("local sockets denied: <OS error>"). The helper also closes a socket it created but could not bind.

Evidence: I ran the test module at this head (29 tests OK), including the new skip-path test, which runs under a patched PermissionError.

Not done: replaying the whole class with a global socket patch. That needed an inline python3 -c, which the rereview command table refused ([layer=toolkit_hook], rule command_table). It was an extra confirmation, not a required check.

Accumulated scope: I reviewed the whole origin/main...7eebf0e diff (merge base f66b2ba, 202 lines). Changed files: scripts/native_prerequisites.py, tests/test_native_prerequisites.py, tests/test_ci_policy.py, docs/dependency-preparation.md, docs/capability-parity-audit.md. Since the last review, only tests/test_native_prerequisites.py changed (commit 7eebf0e). I read both callers again: scripts/executors.py:1271 (bwrap → functional_check and static_only) and :1346 (srt → preparation_isolation, status: unverified, preflight text added to the task).

Acceptance criteria

AC State Evidence
AC-1 covered (project context); not verified in the no-network Codex sandbox All 5 PreflightTests pass, or skip with the OS cause. InstalledCycleTests module: 18 OK. The no-network Codex run is unverified
AC-2 verified test_ci_policy.py: 54 OK. HeadMovedTests uses a disposable repository and a local bare origin
AC-3 covered (project context); not verified in the no-network Codex sandbox The listener denial no longer causes an AttributeError. A denied probe returns unavailable evidence instead of raising
AC-4 verified (unit) test_socket_creation_failure_is_reported_as_unverified. executors.py records the evidence string in native_sandbox_preflight
AC-5 verified docs/dependency-preparation.md and PA-46 (real limit) in docs/capability-parity-audit.md. validate-package.py is VALID

Verification (tests.version 2, head 7eebf0e018f43524b82a9a32206572d671c26590, executed in this rereview)

ID Context Result Evidence
configured-test project pass python3 -m unittest discover -s tests: 2642 tests, 1 skipped, exit 0
ac-1-preflight-tests project pass ... -p test_native_prerequisites.py: 29 tests, exit 0
ac-1-installed-cycle-tests project pass ... -p test_installed_cycle.py: 18 tests, exit 0
ac-2-head-moved-tests project pass ... -p test_ci_policy.py: 54 tests, exit 0
ac-1-3-no-network-codex-suite no-network-codex-sandbox unverified Not run. The known nested Codex limitation means a Claude -p stage cannot start a Codex sandbox, and no host-side capability was supplied
verify:python3 scripts/validate-package.py project pass VALID: 16 skills ..., exit 0
verify:python3 -m unittest project pass exit 0, 0 tests discovered
verify:basedpyright project pass 0 errors, 0 warnings, 0 notes

Conclusion: not_verified, because the required no-network Codex check is unverified.

Security triage: the mandatory minimum matched no changed path. The audit ran anyway, because this change touches the sandbox isolation probe and the plan sets security_required. cc-security-review ran on the full diff and found no vulnerabilities and no potential risks.

  • The probe fails closed: a denied listener returns False, and its callers degrade to static_only or unverified.
  • TemporaryDirectory still removes the synthetic credential files on the early return.
  • The evidence carries only the OS error text.
  • The test fixtures run git with list argv inside temporary directories.

CI (parallel policy, read once): passed: Classify the change, Skills, manifests, and Bash installer, Windows test shard (windows-2). Pending: Windows test shard (windows-1), Windows test shard (windows-3), Windows test shard (windows-4).

Residual risk: the suite has not yet been run in the real no-network Codex sandbox. Other socket users in the suite, such as AF_UNIX binds, were not exercised under that denial.

Stage receipts

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant