Skip to content

The full suite fails in a no-network Codex sandbox: the srt isolation probe crashes on socket creation and one test needs the real origin #508

Closed

Description

@datacas

Problem

The configured full suite (python3 -m unittest discover -s tests) cannot pass inside a Codex sandbox that has no network, and part of the failure is a toolkit defect, not an environment limit. Seen on the reviews of PR #502 (twice: the rereview of 28383d37 and of d3d4815a): the agent had to rerun the suite outside the sandbox to get a real result. On the same head the suite passes on the host (2640 tests, OK) and CI is green.

Reproduction (this host, codex-cli 0.169.0)

codex sandbox --permission-profile code_cycle_write with extends=":workspace" and no network, then python3 -m unittest discover -s tests: 2640 tests, 14 failures, 8 errors, 1 skipped, 1457 s (host: 528 s).

Why the sandbox blocks it: with network disabled, socket.socket(AF_INET) and an AF_UNIX bind fail with Operation not permitted. With network enabled they pass.

Three causes

Tests Count Cause
test_native_prerequisites.PreflightTests 5 scripts/native_prerequisites.py:284 (isolation_probe) opens a listener with socket.socket() and does not handle PermissionError.
test_installed_cycle.InstalledCycleTests 16 The same line, hit in production code: executors.py:1346 calls native.isolation_probe("srt", …) during dispatch, the PermissionError is uncaught, and the whole stage dies with a traceback. No stage rows, no agent.pid; the tests then see empty lists.
test_ci_policy.HeadMovedTests.test_the_new_head_is_synchronized_and_recorded_before_rereview 1 Not hermetic: it runs the stage with cwd=str(ROOT), so the read-only ref audit runs git ls-remote --heads origin against the real GitHub origin of the checkout (publication_audit.py:234, exit 128 without network). It already failed once in CI on PR #502 (run 44046307280) with the same message, "could not capture the read-only ref audit before dispatch", and passed on rerun.

Why this is the toolkit's problem (AGENTS.md, agent capability parity, #450)

  • A limit of the agent's sandbox must be reported and routed around, never turn into a crash. The probe should return not_verified with the cause ("the sandbox forbids creating sockets") and the stage should continue or stop with that reason, as it does for the other prerequisite failures.
  • Whether the user wants a profile that can create local sockets for verification is a consent question, as docker_host is. Today there is no such grant, so the only way out is the agent leaving the sandbox by itself.
  • A test that needs GitHub makes CI flaky and makes every review depend on the network.
  • Precedent: Full test suite cannot pass inside the Claude review sandbox (62 test_validate_package errors, 1 test_executors failure) #274 (same symptom for the Claude review sandbox) was fixed in the tests without changing the sandbox.

Fix direction

  1. native_prerequisites.isolation_probe: catch OSError when creating the listener and return a not_verified result with the real cause; the dispatch must not raise. not_verified is never reported as a pass and never hides a real failure.
  2. Make HeadMovedTests hermetic: use a temporary repository with a local bare origin (or inject a ref auditor) instead of ROOT.
  3. Tests: the probe with socket.socket patched to raise PermissionError; the dispatch path under that condition; HeadMovedTests with no network.
  4. Add the rows to docs/capability-parity-audit.md and update the page that owns the probe (docs/host-permissions.md or the native prerequisites page).

Out of scope (moved to a follow-up issue)

  • Whether the isolation probe can avoid a TCP listener (note: the sandbox also blocks AF_UNIX bind without network, so a Unix socket is not a guaranteed alternative).
  • A consent grant for a verification profile that allows local sockets or network, recorded like docker_host.

Related: #274, #345, #450, #507, PR #502.

Acceptance criteria

  • AC-1: inside a no-network Codex sandbox the 5 PreflightTests and the 16 InstalledCycleTests pass or skip with an explicit environment cause, and no dispatch ends in a traceback.
  • AC-2: HeadMovedTests passes with no network and no real origin.
  • AC-3: the suite run in that sandbox reports every remaining failure as a product failure or as a named environment limit, never as an unexplained crash.
  • AC-4: the stage record says when isolation could not be verified and why.
  • AC-5: documentation and the parity audit are updated.

Activity

  1. added
    bugSomething isn't working
    priority: urgentNeeds attention before other work
    P0Critical path of CCT 2.0
    on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P0Critical path of CCT 2.0bugSomething isn't workingpriority: urgentNeeds attention before other work

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions