fix(runner): name why a file failed when nvim exits non-zero without a reason

scripts/README.md

@@ -63,7 +63,7 @@ The Lua repros run from the repository root with `nvim --headless -u NONE -l scr
 ## Invariants
 
 - `repro_issue_238.sh` and `repro_issue_248.sh` find the server port from the lock file name (`PORT.lock`) in a lock directory, and send the lock file's `authToken` in the `x-claude-code-ide-authorization` header. `repro_issue_277.sh` sends the same header, but reads the port and the token from the plugin state through nvim RPC [repro_issue_238.sh:TOKEN] [repro_issue_248.sh:TOKEN] [repro_issue_277.sh:TOKEN]. All three put the URL before `--header`, because websocat 1.14.1 reads the URL as a header value when `--header` comes first and exits with `No URL specified` [repro_issue_238.sh:WEBSOCAT] [repro_issue_277.sh:WEBSOCAT].
-- `run_integration_tests_individually.sh` must run from the repo root: `-u tests/minimal_init.lua` and `CLAUDECODE_MINIMAL_INIT` (`$PWD/tests/minimal_init.lua`) name the file from the working folder. It reads only the three summary lines that plenary prints, `Success:`, `Failed :` and `Errors :`, and none of the `Success ||` or `Fail ||` rows. A line counts only when it starts with its word, after the color codes are removed, and the last such line of each word wins. It adds a file to `FAILED_FILES` when the summary counts a failure or an error, when nvim exits nonzero, or when the output has no plenary summary: one of the three lines is missing, or its count is not `0` or 1 to 9 of the ASCII digits `0` to `9` with no leading zero, in every locale. Plenary prints the summary only after it ran the whole file, so a missing summary means nvim stopped early, for example in a Lua crash that exits 0. Plenary also prints none for a file without a top-level `describe`. The script prints `FAILED: <file>: no plenary summary (nvim exit N)` for such a file and counts it as one error. `Total Success`, `Total Failed` and `Total Errors` are the sums of the summary counts. It counts a timeout (exit 124), a SIGKILL from `timeout -k 5s` (exit 137) and SIGTERM (exit 143) too, even when every spec passed and plenary printed its summary, because nvim did not exit in time. It prints a `TIMEOUT:` line for each such file and then kills every process left in the process group of `timeout`, since a spec that spins in Lua ignores SIGTERM and leaves its nvim running. It exits 1 when `FAILED_FILES` is non-empty or the parsed failure or error totals are non-zero, else 0 [run_integration_tests_individually.sh:FAILED_FILES, TOTAL_FAILED].
+- `run_integration_tests_individually.sh` must run from the repo root: `-u tests/minimal_init.lua` and `CLAUDECODE_MINIMAL_INIT` (`$PWD/tests/minimal_init.lua`) name the file from the working folder. It reads only the three summary lines that plenary prints, `Success:`, `Failed :` and `Errors :`, and none of the `Success ||` or `Fail ||` rows. A line counts only when it starts with its word, after the color codes are removed, and the last such line of each word wins. It adds a file to `FAILED_FILES` when the summary counts a failure or an error, when nvim or `tee` exits nonzero, or when the output has no plenary summary: one of the three lines is missing, or its count is not `0` or 1 to 9 of the ASCII digits `0` to `9` with no leading zero, in every locale. Plenary prints the summary only after it ran the whole file, so a missing summary means nvim stopped early, for example in a Lua crash that exits 0. Plenary also prints none for a file without a top-level `describe`. The script prints `FAILED: <file>: no plenary summary (nvim exit N)` for such a file and counts it as one error. It adds the signal name after `N` (`nvim exit 139, SIGSEGV`) when `N` is above 128. `Total Success`, `Total Failed` and `Total Errors` are the sums of the summary counts. It counts a timeout (exit 124), a SIGKILL from `timeout -k 5s` (exit 137) and SIGTERM (exit 143) too, even when every spec passed and plenary printed its summary, because nvim did not exit in time. It prints a `TIMEOUT:` line for each such file. The line for exit 137 names `SIGKILL` and ends with "or another process killed it", because the out-of-memory killer gives 137 too. The lines for 124 and 143 end with "or an outside signal stopped it", because an outside SIGTERM that reaches `timeout` gives 124 with uutils and 143 with GNU. A file with no summary that a timeout or a signal stopped gets both the `FAILED:` and the `TIMEOUT:` line. When nvim exits nonzero the script kills every process left in the process group of `timeout`, since a spec that spins in Lua ignores SIGTERM and leaves its nvim running, and since the nvim that plenary starts for a spec runs on when the nvim the script starts dies. When nvim exits nonzero and the script printed neither a `FAILED:` nor a `TIMEOUT:` line for the file, for example after a clean summary, it prints `FAILED: <file>: nvim exit N`, with the signal name when `N` is above 128. When `tee` exits nonzero it prints `FAILED: <file>: tee exit N` too, with the signal name when `N` is above 128, so that a failure of `tee` is not taken for one of nvim. It exits 1 when `FAILED_FILES` is non-empty or the parsed failure or error totals are non-zero, else 0 [run_integration_tests_individually.sh:FAILED_FILES, TOTAL_FAILED].
 - `repro_issue_238.sh` and `repro_issue_248.sh` take the first lock file found, not the newest [repro_issue_238.sh:LOCK_FILE] [repro_issue_248.sh:LOCK_FILE].
 - `repro_issue_238.sh` and `repro_issue_248.sh` read `CLAUDE_LOCKFILE_DIR` from the environment. `repro_issue_248.sh` falls back to `$HOME/.claude/ide` and ignores `CLAUDE_CONFIG_DIR` [repro_issue_238.sh:LOCK_DIR] [repro_issue_248.sh:LOCK_DIR].
 - `manual_test_helper.lua` opens the `README.md` in the repo root that holds the script, found from the script's own path when it loads (the cwd when the chunk has no file name), and returns early when that file is not readable [manual_test_helper.lua:repo_root, readme_path].
@@ -84,7 +84,7 @@ The Lua repros run from the repository root with `nvim --headless -u NONE -l scr
 
 - CAUTION: Do not run the token-sending clients (`repro_issue_238.sh`, `repro_issue_248.sh`, and `repro_issue_277.sh`) on a shared machine. They pass the auth token on the `websocat` command line (`--header`), where other users can read it with `ps` [repro_issue_238.sh:WEBSOCAT] [repro_issue_248.sh:TOKEN] [repro_issue_277.sh:TOKEN].
 - `repro_issue_238.sh`, `repro_issue_248.sh`, and `repro_issue_277.sh` connect to the plugin server on `127.0.0.1` [repro_issue_238.sh:PORT] [repro_issue_248.sh:PORT] [repro_issue_277.sh:PORT].
-- `run_integration_tests_individually.sh` creates two `mktemp` files and two `mktemp -d` folders per test (the output, the file that holds the process id of the file's `timeout`, and the `XDG_STATE_HOME` and `XDG_CONFIG_HOME` that the file's two Neovims share) and deletes all four after the file ran. When it gets SIGTERM, SIGHUP or SIGINT (Ctrl-C) while a file runs, it kills that `timeout`, then the process group of the file, deletes the four and dies by that signal at once, so a calling shell sees status 143, 129 or 130, and a shell loop around the runner stops after Ctrl-C. Killing the `timeout` first covers the moments before it has made its own group, when no Neovim runs yet. Both kills use KILL, so a Neovim that ignores SIGTERM stops too. Two windows of a few milliseconds are not covered. (1) While the runner makes its four items it has not stored their names yet: a signal then leaves the items it already made, never deletes them, and starts no Neovim. (2) Between the moment the runner stores the names and the moment the file's shell has written the process id, there is no `timeout` to kill. A SIGTERM or SIGHUP sent to the runner alone leaves that shell running, and so does any SIGINT (Ctrl-C too), because a shell that runs in the background ignores SIGINT. The Neovim then starts and runs until its own timeout, and two of the four come back after the runner deleted them: the process id file, which the file's shell writes again, and the state folder, which the Neovim makes again. A SIGTERM or SIGHUP sent to the whole process group stops that shell too, so nothing starts and nothing is left. It sets `VIMINIT`, `CLAUDECODE_MINIMAL_INIT`, `XDG_STATE_HOME` and `XDG_CONFIG_HOME` for that file's Neovims and the processes they start only, as a prefix of one command, and leaves `XDG_DATA_HOME` alone because plenary lives there. A run writes nothing under `~/.local/state/nvim`, and the developer's `ftplugin` and `after` files under `~/.config/nvim` do not run [run_integration_tests_individually.sh:temp_output, group_file, state_dir, config_dir].
+- `run_integration_tests_individually.sh` creates two `mktemp` files and two `mktemp -d` folders per test (the output, the file that holds the process id of the file's `timeout`, and the `XDG_STATE_HOME` and `XDG_CONFIG_HOME` that the file's two Neovims share) and deletes all four after the file ran. The file's subshell writes the exit statuses of nvim and of `tee` to a file in the state folder when the file ends, and a write that comes after the cleanup finds the folder gone and fails. When it gets SIGTERM, SIGHUP or SIGINT (Ctrl-C) while a file runs, it kills that `timeout`, then the process group of the file, deletes the four and dies by that signal at once, so a calling shell sees status 143, 129 or 130, and a shell loop around the runner stops after Ctrl-C. Killing the `timeout` first covers the moments before it has made its own group, when no Neovim runs yet. Both kills use KILL, so a Neovim that ignores SIGTERM stops too. Two windows of a few milliseconds are not covered. (1) While the runner makes its four items it has not stored their names yet: a signal then leaves the items it already made, never deletes them, and starts no Neovim. (2) Between the moment the runner stores the names and the moment the file's shell has written the process id, there is no `timeout` to kill. A SIGTERM or SIGHUP sent to the runner alone leaves that shell running, and so does any SIGINT (Ctrl-C in a shell, for one), because a shell that runs in the background ignores SIGINT. The Neovim then starts and runs until its own timeout, and two of the four come back after the runner deleted them: the process id file, which the file's shell writes again, and the state folder, which the Neovim makes again. A SIGTERM or SIGHUP sent to the whole process group stops that shell too, so nothing starts and nothing is left. The same holds when SIGTERM, SIGHUP, SIGINT or SIGUSR1 kills a runner that leads its own session on a terminal, for example one that a terminal emulator starts directly: its death hangs up the foreground process group of the terminal, so that shell dies too. SIGKILL gives the runner no time to clean up: a runner that leads such a session and gets it starts no Neovim, but all four items stay, and without a terminal the Neovim starts as well. Bash ignores SIGQUIT, so the file runs to its end. It sets `VIMINIT`, `CLAUDECODE_MINIMAL_INIT`, `XDG_STATE_HOME` and `XDG_CONFIG_HOME` for that file's Neovims and the processes they start only, as a prefix of one command, and leaves `XDG_DATA_HOME` alone because plenary lives there. A run writes nothing under `~/.local/state/nvim`, and the developer's `ftplugin` and `after` files under `~/.config/nvim` do not run [run_integration_tests_individually.sh:temp_output, group_file, state_dir, config_dir].
 - `repro_issue_231.lua` deletes `<repo_root>/__issue_231_repro__.md` [repro_issue_231.lua:try_open_diff].
 - `repro_issue_70.sh` and `repro_issue_277.sh` make a work dir with `mktemp -d` (`${TMPDIR:-/tmp}/issue70.XXXXXX` or `issue277.XXXXXX`) and an `att-home` folder in it before they check their tools. They install `trap cleanup EXIT` after that check, so an exit 3 for a missing tool leaves the work dir behind. The trap also runs on SIGTERM, SIGINT and SIGHUP. It destroys the `agent-tty` sessions and removes the work dir. agent-tty still leaves an empty socket folder `/tmp/agent-tty/<8 hex digits>` (it ignores `TMPDIR`) and an entry for the deleted home in `${XDG_STATE_HOME:-$HOME/.local/state}/agent-tty/homes.json`, one of each per run. The sessions inherit the caller's environment, so `CLAUDE_CONFIG_DIR` and the `XDG_*` variables reach the Neovim and the `claude` CLI that they run. `repro_issue_70.sh` launches the real `claude` CLI, and it also runs `claude --version` once outside agent-tty. Its cleanup runs `pkill -f "repro_issue_70_probe.lua $REPO_ROOT"`, which also kills the probe of another run in the same checkout [repro_issue_70.sh:WORK, cleanup] [repro_issue_277.sh:WORK, cleanup].
 - `repro_issue_70_probe.lua` starts the real server with `auto_start = true`, so the plugin writes a real `PORT.lock` into the user's lock directory. Other Claude CLIs can see that lock file until `claudecode.stop()` removes it at the end. When the removal fails, `stop` only logs a warning, so a lock file that the removal failed to delete stays [repro_issue_70_probe.lua:claudecode.setup, claudecode.stop] [lua/claudecode/init.lua:M.stop].

scripts/run_integration_tests_individually.sh

@@ -44,6 +44,14 @@ TOTAL_FAILED=0
 TOTAL_ERRORS=0
 FAILED_FILES=()
 
+# The signal behind an exit status above 128, as ", SIGSEGV". Nothing for any other status.
+signal_note() {
+  local name
+  if [[ $1 -gt 128 ]] && name=$(kill -l $(($1 - 128)) 2>/dev/null) && [[ -n $name ]]; then
+    printf ', SIG%s' "$name"
+  fi
+}
+
 # Function to run a single test file
 run_test_file() {
   local test_file=$1
@@ -53,11 +61,12 @@ run_test_file() {
   echo ""
   echo "Running: $basename"
 
-  local temp_output group_file state_dir config_dir
+  local temp_output group_file state_dir config_dir status_file
   temp_output=$(mktemp)
   group_file=$(mktemp)
   state_dir=$(mktemp -d)
   config_dir=$(mktemp -d)
+  status_file=$state_dir/exit_statuses
   CURRENT_TEMP=("$temp_output" "$group_file" "$state_dir" "$config_dir")
   CURRENT_GROUP_FILE=$group_file
 
@@ -70,25 +79,35 @@ run_test_file() {
   # the developer's ftplugin and after files. plenary lives under XDG_DATA_HOME, which stays. Both Neovims share the
   # folders, so the one started here takes -i NONE: two ShaDa writers made the first fail with E886 after a timeout.
   # The file runs in the background and the runner waits for it: `wait` ends at once when the INT trap runs. The
-  # subshell has no pipefail and exits with the status of the first command itself: with pipefail, bash prints a job
-  # report ("Killed", two lines) in the subshell when the cleanup above kills `timeout`.
+  # subshell has no pipefail, because with it bash prints a job report ("Killed", two lines) when the cleanup above
+  # kills `timeout`. It writes the exit statuses of nvim and of `tee` to a file in the state folder, so that the reason
+  # line can name the stage that failed, and exits with the status of nvim, or with that of `tee` when nvim exited 0.
+  # A subshell that ends after the cleanup above finds the folder gone, and its write fails.
   (
     set +o pipefail
+    set +e
     VIMINIT='lua local f = vim.env.CLAUDECODE_MINIMAL_INIT if f then dofile(f) end' \
       CLAUDECODE_MINIMAL_INIT="$PWD/tests/minimal_init.lua" \
       XDG_STATE_HOME="$state_dir" XDG_CONFIG_HOME="$config_dir" \
       bash -c 'echo $$ > "$1"; shift; exec timeout -k 5s "$@"' _ "$group_file" "$FILE_TIMEOUT" \
       nvim --headless -i NONE -u tests/minimal_init.lua \
       -c "lua require('plenary.test_harness').test_file('$test_file')" \
       2>&1 | tee "$temp_output"
-    exit "${PIPESTATUS[0]}"
+    pipe_status=("${PIPESTATUS[@]}")
+    echo "${pipe_status[0]} ${pipe_status[1]}" 2>/dev/null >"$status_file"
+    exit $((pipe_status[0] != 0 ? pipe_status[0] : pipe_status[1]))
   ) &
   if wait $!; then
     EXIT_CODE=0
   else
     EXIT_CODE=$?
   fi
   CURRENT_GROUP_FILE=""
+  local tee_status=0 status_line
+  if status_line=$(cat "$status_file" 2>/dev/null) && [[ $status_line =~ ^([0-9]+)\ ([0-9]+)$ ]]; then
+    EXIT_CODE=${BASH_REMATCH[1]}
+    tee_status=${BASH_REMATCH[2]}
+  fi
 
   # Parse results from output
   local clean_output
@@ -102,8 +121,10 @@ run_test_file() {
   # whatever its exit code says. A count that is not 0 or 1 to 9 digits without a leading zero would stop bash or wrap
   # in the arithmetic below. The digits are spelled out because [0-9] also matches other digits in some locales.
   local plain_count='^(0|[123456789][0123456789]{0,8})$'
+  local reason_printed=false
   if [[ ! $success_count =~ $plain_count ]] || [[ ! $failed_count =~ $plain_count ]] || [[ ! $error_count =~ $plain_count ]]; then
-    echo "FAILED: $basename: no plenary summary (nvim exit $EXIT_CODE)"
+    echo "FAILED: $basename: no plenary summary (nvim exit $EXIT_CODE$(signal_note "$EXIT_CODE"))"
+    reason_printed=true
     success_count=0
     failed_count=0
     error_count=1
@@ -115,14 +136,36 @@ run_test_file() {
   TOTAL_ERRORS=$((TOTAL_ERRORS + error_count))
 
   # Exit 124 is timeout's, 137 is its SIGKILL after 5 more seconds, and 143 is SIGTERM.
+  # An outside SIGTERM that reaches timeout gives 124 (uutils) or 143 (GNU), and the out-of-memory killer gives 137 like
+  # timeout's own KILL, so these lines do not claim that the limit was hit.
   if [[ $EXIT_CODE -eq 124 ]] || [[ $EXIT_CODE -eq 137 ]] || [[ $EXIT_CODE -eq 143 ]]; then
-    echo "TIMEOUT: $basename was stopped (exit $EXIT_CODE): nvim did not exit within $FILE_TIMEOUT"
-    # nvim busy in a Lua loop ignores SIGTERM and outlives timeout.
+    if [[ $EXIT_CODE -eq 137 ]]; then
+      echo "TIMEOUT: $basename was stopped (exit 137, SIGKILL): nvim did not exit within $FILE_TIMEOUT," \
+        "or another process killed it"
+    else
+      echo "TIMEOUT: $basename was stopped (exit $EXIT_CODE): nvim did not exit within $FILE_TIMEOUT," \
+        "or an outside signal stopped it"
+    fi
+    reason_printed=true
+  fi
+
+  # nvim busy in a Lua loop ignores SIGTERM and outlives timeout, and when the nvim started here died, the one that
+  # plenary started for the spec runs on.
+  if [[ $EXIT_CODE -ne 0 ]]; then
     kill -KILL -- "-$(cat "$group_file")" 2>/dev/null || true
   fi
 
+  # A file whose nvim exited non-zero gets a line with a reason: the no-summary line, the TIMEOUT line, or this one. A
+  # file with no summary that a timeout or a signal stopped gets the first two, as on main.
+  if [[ $EXIT_CODE -ne 0 ]] && [[ $reason_printed == false ]]; then
+    echo "FAILED: $basename: nvim exit $EXIT_CODE$(signal_note "$EXIT_CODE")"
+  fi
+  if [[ $tee_status -ne 0 ]]; then
+    echo "FAILED: $basename: tee exit $tee_status$(signal_note "$tee_status")"
+  fi
+
   # Check if test failed
-  if [[ $failed_count -gt 0 ]] || [[ $error_count -gt 0 ]] || [[ $EXIT_CODE -ne 0 ]]; then
+  if [[ $failed_count -gt 0 ]] || [[ $error_count -gt 0 ]] || [[ $EXIT_CODE -ne 0 ]] || [[ $tee_status -ne 0 ]]; then
     FAILED_FILES+=("$basename")
   fi