Repository navigation
Description
Problem
The configured full suite (python3 -m unittest discover -s tests) cannot pass inside a Codex sandbox that has no network, and part of the failure is a toolkit defect, not an environment limit. Seen on the reviews of PR #502 (twice: the rereview of 28383d37 and of d3d4815a): the agent had to rerun the suite outside the sandbox to get a real result. On the same head the suite passes on the host (2640 tests, OK) and CI is green.
Reproduction (this host, codex-cli 0.169.0)
codex sandbox --permission-profile code_cycle_write with extends=":workspace" and no network, then python3 -m unittest discover -s tests: 2640 tests, 14 failures, 8 errors, 1 skipped, 1457 s (host: 528 s).
Why the sandbox blocks it: with network disabled, socket.socket(AF_INET) and an AF_UNIX bind fail with Operation not permitted. With network enabled they pass.
Three causes
| Tests | Count | Cause |
|---|---|---|
test_native_prerequisites.PreflightTests |
5 | scripts/native_prerequisites.py:284 (isolation_probe) opens a listener with socket.socket() and does not handle PermissionError. |
test_installed_cycle.InstalledCycleTests |
16 | The same line, hit in production code: executors.py:1346 calls native.isolation_probe("srt", …) during dispatch, the PermissionError is uncaught, and the whole stage dies with a traceback. No stage rows, no agent.pid; the tests then see empty lists. |
test_ci_policy.HeadMovedTests.test_the_new_head_is_synchronized_and_recorded_before_rereview |
1 | Not hermetic: it runs the stage with cwd=str(ROOT), so the read-only ref audit runs git ls-remote --heads origin against the real GitHub origin of the checkout (publication_audit.py:234, exit 128 without network). It already failed once in CI on PR #502 (run 44046307280) with the same message, "could not capture the read-only ref audit before dispatch", and passed on rerun. |
Why this is the toolkit's problem (AGENTS.md, agent capability parity, #450)
- A limit of the agent's sandbox must be reported and routed around, never turn into a crash. The probe should return
not_verifiedwith the cause ("the sandbox forbids creating sockets") and the stage should continue or stop with that reason, as it does for the other prerequisite failures. - Whether the user wants a profile that can create local sockets for verification is a consent question, as
docker_hostis. Today there is no such grant, so the only way out is the agent leaving the sandbox by itself. - A test that needs GitHub makes CI flaky and makes every review depend on the network.
- Precedent: Full test suite cannot pass inside the Claude review sandbox (62 test_validate_package errors, 1 test_executors failure) #274 (same symptom for the Claude review sandbox) was fixed in the tests without changing the sandbox.
Fix direction
native_prerequisites.isolation_probe: catchOSErrorwhen creating the listener and return anot_verifiedresult with the real cause; the dispatch must not raise.not_verifiedis never reported as a pass and never hides a real failure.- Make
HeadMovedTestshermetic: use a temporary repository with a local bareorigin(or inject a ref auditor) instead ofROOT. - Tests: the probe with
socket.socketpatched to raisePermissionError; the dispatch path under that condition;HeadMovedTestswith no network. - Add the rows to
docs/capability-parity-audit.mdand update the page that owns the probe (docs/host-permissions.mdor the native prerequisites page).
Out of scope (moved to a follow-up issue)
- Whether the isolation probe can avoid a TCP listener (note: the sandbox also blocks
AF_UNIX bindwithout network, so a Unix socket is not a guaranteed alternative). - A consent grant for a verification profile that allows local sockets or network, recorded like
docker_host.
Related: #274, #345, #450, #507, PR #502.
Acceptance criteria
- AC-1: inside a no-network Codex sandbox the 5
PreflightTestsand the 16InstalledCycleTestspass or skip with an explicit environment cause, and no dispatch ends in a traceback. - AC-2:
HeadMovedTestspasses with no network and no realorigin. - AC-3: the suite run in that sandbox reports every remaining failure as a product failure or as a named environment limit, never as an unexplained crash.
- AC-4: the stage record says when isolation could not be verified and why.
- AC-5: documentation and the parity audit are updated.
Activity