From 6e2cc4daa5a676695593a07cf121ea4c3715ead4 Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 12 Nov 2026 06:58:20 +0000 Subject: [PATCH 1/3] fix(runner): name why a file failed when nvim exits non-zero without a reason MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - A file whose nvim exits non-zero gets one line with a reason: the `FAILED: : no plenary summary` line, the `TIMEOUT:` line, or a new `FAILED: : nvim exit N` line. Before, a crash or an exit 1 after a clean summary printed nothing and the file only showed up in the final list. An exit status above 128 names its signal (`nvim exit 139, SIGSEGV`), also in the no-summary line. - The TIMEOUT line for exit 137 names SIGKILL and says "or another process killed it", because the out-of-memory killer gives 137 too. - The `-i NONE` test also checks `--headless` and `-u tests/minimal_init.lua`. - scripts/README.md says that a runner that leads its own session on a terminal leaves nothing after any signal in the early window, and lists the new output lines. - The comment on the subshell says that set -e ends it with the status of `tee` when `tee` fails. _Generated with xum • Model: anthropic:claude-sonnet-5-5 • Thinking: xhigh_ --- scripts/README.md | 4 +- scripts/run_integration_tests_individually.sh | 31 ++++- tests/unit/integration_runner_spec.lua | 115 ++++++++++++++++-- 3 files changed, 137 insertions(+), 13 deletions(-) diff --git a/scripts/README.md b/scripts/README.md index 6ae9bb4f5..26fc9fb65 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -63,7 +63,7 @@ The Lua repros run from the repository root with `nvim --headless -u NONE -l scr ## Invariants - `repro_issue_238.sh` and `repro_issue_248.sh` find the server port from the lock file name (`PORT.lock`) in a lock directory, and send the lock file's `authToken` in the `x-claude-code-ide-authorization` header. `repro_issue_277.sh` sends the same header, but reads the port and the token from the plugin state through nvim RPC [repro_issue_238.sh:TOKEN] [repro_issue_248.sh:TOKEN] [repro_issue_277.sh:TOKEN]. All three put the URL before `--header`, because websocat 1.14.1 reads the URL as a header value when `--header` comes first and exits with `No URL specified` [repro_issue_238.sh:WEBSOCAT] [repro_issue_277.sh:WEBSOCAT]. -- `run_integration_tests_individually.sh` must run from the repo root: `-u tests/minimal_init.lua` and `CLAUDECODE_MINIMAL_INIT` (`$PWD/tests/minimal_init.lua`) name the file from the working folder. It reads only the three summary lines that plenary prints, `Success:`, `Failed :` and `Errors :`, and none of the `Success ||` or `Fail ||` rows. A line counts only when it starts with its word, after the color codes are removed, and the last such line of each word wins. It adds a file to `FAILED_FILES` when the summary counts a failure or an error, when nvim exits nonzero, or when the output has no plenary summary: one of the three lines is missing, or its count is not `0` or 1 to 9 of the ASCII digits `0` to `9` with no leading zero, in every locale. Plenary prints the summary only after it ran the whole file, so a missing summary means nvim stopped early, for example in a Lua crash that exits 0. Plenary also prints none for a file without a top-level `describe`. The script prints `FAILED: : no plenary summary (nvim exit N)` for such a file and counts it as one error. `Total Success`, `Total Failed` and `Total Errors` are the sums of the summary counts. It counts a timeout (exit 124), a SIGKILL from `timeout -k 5s` (exit 137) and SIGTERM (exit 143) too, even when every spec passed and plenary printed its summary, because nvim did not exit in time. It prints a `TIMEOUT:` line for each such file and then kills every process left in the process group of `timeout`, since a spec that spins in Lua ignores SIGTERM and leaves its nvim running. It exits 1 when `FAILED_FILES` is non-empty or the parsed failure or error totals are non-zero, else 0 [run_integration_tests_individually.sh:FAILED_FILES, TOTAL_FAILED]. +- `run_integration_tests_individually.sh` must run from the repo root: `-u tests/minimal_init.lua` and `CLAUDECODE_MINIMAL_INIT` (`$PWD/tests/minimal_init.lua`) name the file from the working folder. It reads only the three summary lines that plenary prints, `Success:`, `Failed :` and `Errors :`, and none of the `Success ||` or `Fail ||` rows. A line counts only when it starts with its word, after the color codes are removed, and the last such line of each word wins. It adds a file to `FAILED_FILES` when the summary counts a failure or an error, when nvim exits nonzero, or when the output has no plenary summary: one of the three lines is missing, or its count is not `0` or 1 to 9 of the ASCII digits `0` to `9` with no leading zero, in every locale. Plenary prints the summary only after it ran the whole file, so a missing summary means nvim stopped early, for example in a Lua crash that exits 0. Plenary also prints none for a file without a top-level `describe`. The script prints `FAILED: : no plenary summary (nvim exit N)` for such a file and counts it as one error. It adds the signal name after `N` (`nvim exit 139, SIGSEGV`) when `N` is above 128. `Total Success`, `Total Failed` and `Total Errors` are the sums of the summary counts. It counts a timeout (exit 124), a SIGKILL from `timeout -k 5s` (exit 137) and SIGTERM (exit 143) too, even when every spec passed and plenary printed its summary, because nvim did not exit in time. It prints a `TIMEOUT:` line for each such file, with `SIGKILL` named for exit 137 and the words "or another process killed it" because the out-of-memory killer gives 137 too. Then it kills every process left in the process group of `timeout`, since a spec that spins in Lua ignores SIGTERM and leaves its nvim running. When nvim exits nonzero and the script printed neither a `FAILED:` nor a `TIMEOUT:` line for the file, for example after a clean summary, it prints `FAILED: : nvim exit N`, with the signal name when `N` is above 128. It exits 1 when `FAILED_FILES` is non-empty or the parsed failure or error totals are non-zero, else 0 [run_integration_tests_individually.sh:FAILED_FILES, TOTAL_FAILED]. - `repro_issue_238.sh` and `repro_issue_248.sh` take the first lock file found, not the newest [repro_issue_238.sh:LOCK_FILE] [repro_issue_248.sh:LOCK_FILE]. - `repro_issue_238.sh` and `repro_issue_248.sh` read `CLAUDE_LOCKFILE_DIR` from the environment. `repro_issue_248.sh` falls back to `$HOME/.claude/ide` and ignores `CLAUDE_CONFIG_DIR` [repro_issue_238.sh:LOCK_DIR] [repro_issue_248.sh:LOCK_DIR]. - `manual_test_helper.lua` opens the `README.md` in the repo root that holds the script, found from the script's own path when it loads (the cwd when the chunk has no file name), and returns early when that file is not readable [manual_test_helper.lua:repo_root, readme_path]. @@ -84,7 +84,7 @@ The Lua repros run from the repository root with `nvim --headless -u NONE -l scr - CAUTION: Do not run the token-sending clients (`repro_issue_238.sh`, `repro_issue_248.sh`, and `repro_issue_277.sh`) on a shared machine. They pass the auth token on the `websocat` command line (`--header`), where other users can read it with `ps` [repro_issue_238.sh:WEBSOCAT] [repro_issue_248.sh:TOKEN] [repro_issue_277.sh:TOKEN]. - `repro_issue_238.sh`, `repro_issue_248.sh`, and `repro_issue_277.sh` connect to the plugin server on `127.0.0.1` [repro_issue_238.sh:PORT] [repro_issue_248.sh:PORT] [repro_issue_277.sh:PORT]. -- `run_integration_tests_individually.sh` creates two `mktemp` files and two `mktemp -d` folders per test (the output, the file that holds the process id of the file's `timeout`, and the `XDG_STATE_HOME` and `XDG_CONFIG_HOME` that the file's two Neovims share) and deletes all four after the file ran. When it gets SIGTERM, SIGHUP or SIGINT (Ctrl-C) while a file runs, it kills that `timeout`, then the process group of the file, deletes the four and dies by that signal at once, so a calling shell sees status 143, 129 or 130, and a shell loop around the runner stops after Ctrl-C. Killing the `timeout` first covers the moments before it has made its own group, when no Neovim runs yet. Both kills use KILL, so a Neovim that ignores SIGTERM stops too. Two windows of a few milliseconds are not covered. (1) While the runner makes its four items it has not stored their names yet: a signal then leaves the items it already made, never deletes them, and starts no Neovim. (2) Between the moment the runner stores the names and the moment the file's shell has written the process id, there is no `timeout` to kill. A SIGTERM or SIGHUP sent to the runner alone leaves that shell running, and so does any SIGINT (Ctrl-C too), because a shell that runs in the background ignores SIGINT. The Neovim then starts and runs until its own timeout, and two of the four come back after the runner deleted them: the process id file, which the file's shell writes again, and the state folder, which the Neovim makes again. A SIGTERM or SIGHUP sent to the whole process group stops that shell too, so nothing starts and nothing is left. It sets `VIMINIT`, `CLAUDECODE_MINIMAL_INIT`, `XDG_STATE_HOME` and `XDG_CONFIG_HOME` for that file's Neovims and the processes they start only, as a prefix of one command, and leaves `XDG_DATA_HOME` alone because plenary lives there. A run writes nothing under `~/.local/state/nvim`, and the developer's `ftplugin` and `after` files under `~/.config/nvim` do not run [run_integration_tests_individually.sh:temp_output, group_file, state_dir, config_dir]. +- `run_integration_tests_individually.sh` creates two `mktemp` files and two `mktemp -d` folders per test (the output, the file that holds the process id of the file's `timeout`, and the `XDG_STATE_HOME` and `XDG_CONFIG_HOME` that the file's two Neovims share) and deletes all four after the file ran. When it gets SIGTERM, SIGHUP or SIGINT (Ctrl-C) while a file runs, it kills that `timeout`, then the process group of the file, deletes the four and dies by that signal at once, so a calling shell sees status 143, 129 or 130, and a shell loop around the runner stops after Ctrl-C. Killing the `timeout` first covers the moments before it has made its own group, when no Neovim runs yet. Both kills use KILL, so a Neovim that ignores SIGTERM stops too. Two windows of a few milliseconds are not covered. (1) While the runner makes its four items it has not stored their names yet: a signal then leaves the items it already made, never deletes them, and starts no Neovim. (2) Between the moment the runner stores the names and the moment the file's shell has written the process id, there is no `timeout` to kill. A SIGTERM or SIGHUP sent to the runner alone leaves that shell running, and so does any SIGINT (Ctrl-C in a shell, for one), because a shell that runs in the background ignores SIGINT. The Neovim then starts and runs until its own timeout, and two of the four come back after the runner deleted them: the process id file, which the file's shell writes again, and the state folder, which the Neovim makes again. A SIGTERM or SIGHUP sent to the whole process group stops that shell too, so nothing starts and nothing is left. The same holds for any signal that kills a runner that leads its own session on a terminal, for example one that a terminal emulator starts directly: its death hangs up the foreground process group of the terminal. It sets `VIMINIT`, `CLAUDECODE_MINIMAL_INIT`, `XDG_STATE_HOME` and `XDG_CONFIG_HOME` for that file's Neovims and the processes they start only, as a prefix of one command, and leaves `XDG_DATA_HOME` alone because plenary lives there. A run writes nothing under `~/.local/state/nvim`, and the developer's `ftplugin` and `after` files under `~/.config/nvim` do not run [run_integration_tests_individually.sh:temp_output, group_file, state_dir, config_dir]. - `repro_issue_231.lua` deletes `/__issue_231_repro__.md` [repro_issue_231.lua:try_open_diff]. - `repro_issue_70.sh` and `repro_issue_277.sh` make a work dir with `mktemp -d` (`${TMPDIR:-/tmp}/issue70.XXXXXX` or `issue277.XXXXXX`) and an `att-home` folder in it before they check their tools. They install `trap cleanup EXIT` after that check, so an exit 3 for a missing tool leaves the work dir behind. The trap also runs on SIGTERM, SIGINT and SIGHUP. It destroys the `agent-tty` sessions and removes the work dir. agent-tty still leaves an empty socket folder `/tmp/agent-tty/<8 hex digits>` (it ignores `TMPDIR`) and an entry for the deleted home in `${XDG_STATE_HOME:-$HOME/.local/state}/agent-tty/homes.json`, one of each per run. The sessions inherit the caller's environment, so `CLAUDE_CONFIG_DIR` and the `XDG_*` variables reach the Neovim and the `claude` CLI that they run. `repro_issue_70.sh` launches the real `claude` CLI, and it also runs `claude --version` once outside agent-tty. Its cleanup runs `pkill -f "repro_issue_70_probe.lua $REPO_ROOT"`, which also kills the probe of another run in the same checkout [repro_issue_70.sh:WORK, cleanup] [repro_issue_277.sh:WORK, cleanup]. - `repro_issue_70_probe.lua` starts the real server with `auto_start = true`, so the plugin writes a real `PORT.lock` into the user's lock directory. Other Claude CLIs can see that lock file until `claudecode.stop()` removes it at the end. When the removal fails, `stop` only logs a warning, so a lock file that the removal failed to delete stays [repro_issue_70_probe.lua:claudecode.setup, claudecode.stop] [lua/claudecode/init.lua:M.stop]. diff --git a/scripts/run_integration_tests_individually.sh b/scripts/run_integration_tests_individually.sh index 044ce57a8..f2e7324af 100755 --- a/scripts/run_integration_tests_individually.sh +++ b/scripts/run_integration_tests_individually.sh @@ -44,6 +44,14 @@ TOTAL_FAILED=0 TOTAL_ERRORS=0 FAILED_FILES=() +# The signal behind an exit status above 128, as ", SIGSEGV". Nothing for any other status. +signal_note() { + local name + if [[ $1 -gt 128 ]] && name=$(kill -l $(($1 - 128)) 2>/dev/null) && [[ -n $name ]]; then + printf ', SIG%s' "$name" + fi +} + # Function to run a single test file run_test_file() { local test_file=$1 @@ -70,8 +78,9 @@ run_test_file() { # the developer's ftplugin and after files. plenary lives under XDG_DATA_HOME, which stays. Both Neovims share the # folders, so the one started here takes -i NONE: two ShaDa writers made the first fail with E886 after a timeout. # The file runs in the background and the runner waits for it: `wait` ends at once when the INT trap runs. The - # subshell has no pipefail and exits with the status of the first command itself: with pipefail, bash prints a job - # report ("Killed", two lines) in the subshell when the cleanup above kills `timeout`. + # subshell has no pipefail, because with it bash prints a job report ("Killed", two lines) when the cleanup above + # kills `timeout`. It exits with the status of the first command. When `tee` fails, set -e ends the subshell with + # the status of `tee`, as pipefail did. ( set +o pipefail VIMINIT='lua local f = vim.env.CLAUDECODE_MINIMAL_INIT if f then dofile(f) end' \ @@ -102,8 +111,10 @@ run_test_file() { # whatever its exit code says. A count that is not 0 or 1 to 9 digits without a leading zero would stop bash or wrap # in the arithmetic below. The digits are spelled out because [0-9] also matches other digits in some locales. local plain_count='^(0|[123456789][0123456789]{0,8})$' + local reason_printed=false if [[ ! $success_count =~ $plain_count ]] || [[ ! $failed_count =~ $plain_count ]] || [[ ! $error_count =~ $plain_count ]]; then - echo "FAILED: $basename: no plenary summary (nvim exit $EXIT_CODE)" + echo "FAILED: $basename: no plenary summary (nvim exit $EXIT_CODE$(signal_note "$EXIT_CODE"))" + reason_printed=true success_count=0 failed_count=0 error_count=1 @@ -116,11 +127,23 @@ run_test_file() { # Exit 124 is timeout's, 137 is its SIGKILL after 5 more seconds, and 143 is SIGTERM. if [[ $EXIT_CODE -eq 124 ]] || [[ $EXIT_CODE -eq 137 ]] || [[ $EXIT_CODE -eq 143 ]]; then - echo "TIMEOUT: $basename was stopped (exit $EXIT_CODE): nvim did not exit within $FILE_TIMEOUT" + if [[ $EXIT_CODE -eq 137 ]]; then + # The out-of-memory killer gives 137 too, so this line does not claim that the limit was hit. + echo "TIMEOUT: $basename was stopped (exit 137, SIGKILL): nvim did not exit within $FILE_TIMEOUT," \ + "or another process killed it" + else + echo "TIMEOUT: $basename was stopped (exit $EXIT_CODE): nvim did not exit within $FILE_TIMEOUT" + fi + reason_printed=true # nvim busy in a Lua loop ignores SIGTERM and outlives timeout. kill -KILL -- "-$(cat "$group_file")" 2>/dev/null || true fi + # A file that failed with a non-zero exit status always gets one line with a reason: the one above, or this one. + if [[ $EXIT_CODE -ne 0 ]] && [[ $reason_printed == false ]]; then + echo "FAILED: $basename: nvim exit $EXIT_CODE$(signal_note "$EXIT_CODE")" + fi + # Check if test failed if [[ $failed_count -gt 0 ]] || [[ $error_count -gt 0 ]] || [[ $EXIT_CODE -ne 0 ]]; then FAILED_FILES+=("$basename") diff --git a/tests/unit/integration_runner_spec.lua b/tests/unit/integration_runner_spec.lua index 92bb805ef..4a50812d3 100644 --- a/tests/unit/integration_runner_spec.lua +++ b/tests/unit/integration_runner_spec.lua @@ -65,6 +65,7 @@ local STUB_NVIM = concat({ "#!/bin/sh" }, SPEC_NAME, { ' echo $! > "$STUB_DIR/$name.orphan.pid"', "fi", 'cat "$STUB_DIR/$name.out"', + 'if [ -f "$STUB_DIR/$name.die_by" ]; then kill -"$(cat "$STUB_DIR/$name.die_by")" $$; fi', 'exit "$(cat "$STUB_DIR/$name.exit")"', }) @@ -195,7 +196,15 @@ describe("scripts/run_integration_tests_individually.sh", function() temp_dir.remove(dir) end) - ---@param specs { name: string, exit: integer, output: string[], orphan: boolean|nil, ignore_term: boolean|nil }[] + ---@class ScriptSpec + ---@field name string + ---@field exit integer + ---@field output string[] + ---@field orphan boolean|nil + ---@field ignore_term boolean|nil + ---@field die_by string|nil A signal the stub nvim sends itself after its output, as `kill -` takes it + + ---@param specs ScriptSpec[] ---@param locale string|nil The LC_ALL to run the script under ---@return string status ---@return string output @@ -211,6 +220,9 @@ describe("scripts/run_integration_tests_individually.sh", function() if spec.ignore_term then write(dir .. "/" .. spec.name .. ".ignore_term", {}) end + if spec.die_by then + write(dir .. "/" .. spec.name .. ".die_by", { spec.die_by }) + end end local command = "cd " .. quote(dir) @@ -402,13 +414,20 @@ describe("scripts/run_integration_tests_individually.sh", function() for token in args:gmatch("%S+") do table.insert(tokens, token) end - local has_no_shada = false - for index, token in ipairs(tokens) do - if token == "-i" and tokens[index + 1] == "NONE" then - has_no_shada = true + ---@param first string + ---@param second string|nil The token that must follow `first`, or nil when `first` alone is enough + ---@return boolean + local function has(first, second) + for index, token in ipairs(tokens) do + if token == first and (second == nil or tokens[index + 1] == second) then + return true + end end + return false end - assert.is_true(has_no_shada, args) + assert.is_true(has("-i", "NONE"), args) + assert.is_true(has("--headless"), args) + assert.is_true(has("-u", "tests/minimal_init.lua"), args) end) ---@class SignalRun @@ -618,6 +637,88 @@ describe("scripts/run_integration_tests_individually.sh", function() assert.is_nil(output:find("TIMEOUT", 1, true), output) end) + ---@param text string + ---@param needle string + ---@return integer + local function occurrences(text, needle) + local count, from = 0, 1 + while true do + local found, last = text:find(needle, from, true) + if not found then + return count + end + count = count + 1 + from = last + 1 + end + end + + for _, case in ipairs({ + { 139, ", SIGSEGV" }, + { 134, ", SIGABRT" }, + { 1, "" }, + }) do + local exit_status, note = case[1], case[2] + it("names the exit status of a nvim that fails with " .. exit_status .. " after a clean summary", function() + local status, output = run_script({ + { name = "crash_after_spec.lua", exit = exit_status, output = summary(3, 0, 0) }, + }) + + assert.are.equal("1", status, output) + assert.is_truthy( + output:find("FAILED: crash_after_spec.lua: nvim exit " .. exit_status .. note .. "\n", 1, true), + output + ) + assert.are.equal(1, occurrences(output, "FAILED:"), output) + assert.is_truthy(output:find("Total Success: 3", 1, true), output) + assert.is_truthy(output:find("Failed test files:\n - crash_after_spec.lua", 1, true), output) + end) + end + + it("names the signal in the line of a file whose nvim crashed before any summary", function() + local status, output = run_script({ + { name = "crash_spec.lua", exit = 139, output = hung_output("crash_spec.lua") }, + }) + + assert.are.equal("1", status, output) + assert.is_truthy( + output:find("FAILED: crash_spec.lua: no plenary summary (nvim exit 139, SIGSEGV)", 1, true), + output + ) + assert.are.equal(1, occurrences(output, "FAILED:"), output) + assert.is_nil(output:find("TIMEOUT", 1, true), output) + end) + + it("names SIGKILL in the TIMEOUT line of a nvim that was killed, which the limit may not have done", function() + local status, output = run_script({ + { name = "oom_spec.lua", exit = 0, output = hung_output("oom_spec.lua"), die_by = "KILL" }, + }) + + local _, limit = timeout_limits() + assert.are.equal("1", status, output) + assert.is_truthy( + output:find( + "TIMEOUT: oom_spec.lua was stopped (exit 137, SIGKILL): nvim did not exit within " + .. limit + .. "s, or another process killed it", + 1, + true + ), + output + ) + assert.are.equal(1, occurrences(output, "TIMEOUT:"), output) + assert.is_truthy(output:find("Failed test files:\n - oom_spec.lua", 1, true), output) + end) + + it("prints one line with a reason for a nvim that was killed after a clean summary", function() + local status, output = run_script({ + { name = "oom_after_spec.lua", exit = 0, output = summary(2, 0, 0), die_by = "KILL" }, + }) + + assert.are.equal("1", status, output) + assert.is_truthy(output:find("TIMEOUT: oom_after_spec.lua was stopped (exit 137, SIGKILL)", 1, true), output) + assert.is_nil(output:find("FAILED:", 1, true), output) + end) + it("fails a file whose nvim exited 0 after passing tests without printing a summary", function() local status, output = run_script({ { name = "stopped_spec.lua", exit = 0, output = { row("Success", "first"), row("Success", "second") } }, @@ -904,7 +1005,7 @@ describe("scripts/run_integration_tests_individually.sh", function() }) assert.are.equal("1", status, output) - assert.is_truthy(output:find("TIMEOUT: stuck_spec.lua was stopped (exit 137)", 1, true), output) + assert.is_truthy(output:find("TIMEOUT: stuck_spec.lua was stopped (exit 137, SIGKILL)", 1, true), output) assert.is_truthy(output:find("Failed test files:\n - stuck_spec.lua", 1, true), output) end) From 22da0be5feff5c38ca1e0cd0b7950b253a1f41ae Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 12 Nov 2026 09:53:35 +0000 Subject: [PATCH 2/3] fix(runner): name the failing stage, kill what a crashed nvim left, word the TIMEOUT lines MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - The subshell writes the exit statuses of nvim and of `tee` to a status file next to the output file. A failing `tee` gets its own line (`FAILED: : tee exit N`) and no longer shows as an nvim exit. When both fail, both lines print. - The process group of the file is killed after any non-zero nvim exit, not only after a timeout status. The nvim that plenary starts for a spec used to run on when the nvim the runner starts died by a signal. - The TIMEOUT lines for 124 and 143 end with "or an outside signal stopped it": an outside SIGTERM reaches timeout as 124 with uutils and 143 with GNU. - The comment on the reason lines no longer says "always one line": a file with no summary that a timeout or a signal stopped gets two, as on main. - New tests: exit 129 and 130 name SIGHUP and SIGINT, tee stages, the outside SIGTERM with both timeout builds, the group kill after a crash, and that no temp item stays after a failed file. - scripts/README.md says what SIGKILL and SIGQUIT do in the early window of a runner that leads a pty session, and describes the new lines. _Generated with xum • Model: anthropic:claude-sonnet-5-5 • Thinking: xhigh_ --- scripts/README.md | 4 +- scripts/run_integration_tests_individually.sh | 41 +++++-- tests/unit/integration_runner_spec.lua | 103 ++++++++++++++++-- 3 files changed, 128 insertions(+), 20 deletions(-) diff --git a/scripts/README.md b/scripts/README.md index 26fc9fb65..5b8c95eff 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -63,7 +63,7 @@ The Lua repros run from the repository root with `nvim --headless -u NONE -l scr ## Invariants - `repro_issue_238.sh` and `repro_issue_248.sh` find the server port from the lock file name (`PORT.lock`) in a lock directory, and send the lock file's `authToken` in the `x-claude-code-ide-authorization` header. `repro_issue_277.sh` sends the same header, but reads the port and the token from the plugin state through nvim RPC [repro_issue_238.sh:TOKEN] [repro_issue_248.sh:TOKEN] [repro_issue_277.sh:TOKEN]. All three put the URL before `--header`, because websocat 1.14.1 reads the URL as a header value when `--header` comes first and exits with `No URL specified` [repro_issue_238.sh:WEBSOCAT] [repro_issue_277.sh:WEBSOCAT]. -- `run_integration_tests_individually.sh` must run from the repo root: `-u tests/minimal_init.lua` and `CLAUDECODE_MINIMAL_INIT` (`$PWD/tests/minimal_init.lua`) name the file from the working folder. It reads only the three summary lines that plenary prints, `Success:`, `Failed :` and `Errors :`, and none of the `Success ||` or `Fail ||` rows. A line counts only when it starts with its word, after the color codes are removed, and the last such line of each word wins. It adds a file to `FAILED_FILES` when the summary counts a failure or an error, when nvim exits nonzero, or when the output has no plenary summary: one of the three lines is missing, or its count is not `0` or 1 to 9 of the ASCII digits `0` to `9` with no leading zero, in every locale. Plenary prints the summary only after it ran the whole file, so a missing summary means nvim stopped early, for example in a Lua crash that exits 0. Plenary also prints none for a file without a top-level `describe`. The script prints `FAILED: : no plenary summary (nvim exit N)` for such a file and counts it as one error. It adds the signal name after `N` (`nvim exit 139, SIGSEGV`) when `N` is above 128. `Total Success`, `Total Failed` and `Total Errors` are the sums of the summary counts. It counts a timeout (exit 124), a SIGKILL from `timeout -k 5s` (exit 137) and SIGTERM (exit 143) too, even when every spec passed and plenary printed its summary, because nvim did not exit in time. It prints a `TIMEOUT:` line for each such file, with `SIGKILL` named for exit 137 and the words "or another process killed it" because the out-of-memory killer gives 137 too. Then it kills every process left in the process group of `timeout`, since a spec that spins in Lua ignores SIGTERM and leaves its nvim running. When nvim exits nonzero and the script printed neither a `FAILED:` nor a `TIMEOUT:` line for the file, for example after a clean summary, it prints `FAILED: : nvim exit N`, with the signal name when `N` is above 128. It exits 1 when `FAILED_FILES` is non-empty or the parsed failure or error totals are non-zero, else 0 [run_integration_tests_individually.sh:FAILED_FILES, TOTAL_FAILED]. +- `run_integration_tests_individually.sh` must run from the repo root: `-u tests/minimal_init.lua` and `CLAUDECODE_MINIMAL_INIT` (`$PWD/tests/minimal_init.lua`) name the file from the working folder. It reads only the three summary lines that plenary prints, `Success:`, `Failed :` and `Errors :`, and none of the `Success ||` or `Fail ||` rows. A line counts only when it starts with its word, after the color codes are removed, and the last such line of each word wins. It adds a file to `FAILED_FILES` when the summary counts a failure or an error, when nvim or `tee` exits nonzero, or when the output has no plenary summary: one of the three lines is missing, or its count is not `0` or 1 to 9 of the ASCII digits `0` to `9` with no leading zero, in every locale. Plenary prints the summary only after it ran the whole file, so a missing summary means nvim stopped early, for example in a Lua crash that exits 0. Plenary also prints none for a file without a top-level `describe`. The script prints `FAILED: : no plenary summary (nvim exit N)` for such a file and counts it as one error. It adds the signal name after `N` (`nvim exit 139, SIGSEGV`) when `N` is above 128. `Total Success`, `Total Failed` and `Total Errors` are the sums of the summary counts. It counts a timeout (exit 124), a SIGKILL from `timeout -k 5s` (exit 137) and SIGTERM (exit 143) too, even when every spec passed and plenary printed its summary, because nvim did not exit in time. It prints a `TIMEOUT:` line for each such file. The line for exit 137 names `SIGKILL` and ends with "or another process killed it", because the out-of-memory killer gives 137 too. The lines for 124 and 143 end with "or an outside signal stopped it", because an outside SIGTERM that reaches `timeout` gives 124 with uutils and 143 with GNU. A file with no summary that a timeout or a signal stopped gets both the `FAILED:` and the `TIMEOUT:` line. When nvim exits nonzero the script kills every process left in the process group of `timeout`, since a spec that spins in Lua ignores SIGTERM and leaves its nvim running, and since the nvim that plenary starts for a spec runs on when the nvim the script starts dies. When nvim exits nonzero and the script printed neither a `FAILED:` nor a `TIMEOUT:` line for the file, for example after a clean summary, it prints `FAILED: : nvim exit N`, with the signal name when `N` is above 128. When `tee` exits nonzero it prints `FAILED: : tee exit N` too, with the signal name when `N` is above 128, so that a failure of `tee` is not taken for one of nvim. It exits 1 when `FAILED_FILES` is non-empty or the parsed failure or error totals are non-zero, else 0 [run_integration_tests_individually.sh:FAILED_FILES, TOTAL_FAILED]. - `repro_issue_238.sh` and `repro_issue_248.sh` take the first lock file found, not the newest [repro_issue_238.sh:LOCK_FILE] [repro_issue_248.sh:LOCK_FILE]. - `repro_issue_238.sh` and `repro_issue_248.sh` read `CLAUDE_LOCKFILE_DIR` from the environment. `repro_issue_248.sh` falls back to `$HOME/.claude/ide` and ignores `CLAUDE_CONFIG_DIR` [repro_issue_238.sh:LOCK_DIR] [repro_issue_248.sh:LOCK_DIR]. - `manual_test_helper.lua` opens the `README.md` in the repo root that holds the script, found from the script's own path when it loads (the cwd when the chunk has no file name), and returns early when that file is not readable [manual_test_helper.lua:repo_root, readme_path]. @@ -84,7 +84,7 @@ The Lua repros run from the repository root with `nvim --headless -u NONE -l scr - CAUTION: Do not run the token-sending clients (`repro_issue_238.sh`, `repro_issue_248.sh`, and `repro_issue_277.sh`) on a shared machine. They pass the auth token on the `websocat` command line (`--header`), where other users can read it with `ps` [repro_issue_238.sh:WEBSOCAT] [repro_issue_248.sh:TOKEN] [repro_issue_277.sh:TOKEN]. - `repro_issue_238.sh`, `repro_issue_248.sh`, and `repro_issue_277.sh` connect to the plugin server on `127.0.0.1` [repro_issue_238.sh:PORT] [repro_issue_248.sh:PORT] [repro_issue_277.sh:PORT]. -- `run_integration_tests_individually.sh` creates two `mktemp` files and two `mktemp -d` folders per test (the output, the file that holds the process id of the file's `timeout`, and the `XDG_STATE_HOME` and `XDG_CONFIG_HOME` that the file's two Neovims share) and deletes all four after the file ran. When it gets SIGTERM, SIGHUP or SIGINT (Ctrl-C) while a file runs, it kills that `timeout`, then the process group of the file, deletes the four and dies by that signal at once, so a calling shell sees status 143, 129 or 130, and a shell loop around the runner stops after Ctrl-C. Killing the `timeout` first covers the moments before it has made its own group, when no Neovim runs yet. Both kills use KILL, so a Neovim that ignores SIGTERM stops too. Two windows of a few milliseconds are not covered. (1) While the runner makes its four items it has not stored their names yet: a signal then leaves the items it already made, never deletes them, and starts no Neovim. (2) Between the moment the runner stores the names and the moment the file's shell has written the process id, there is no `timeout` to kill. A SIGTERM or SIGHUP sent to the runner alone leaves that shell running, and so does any SIGINT (Ctrl-C in a shell, for one), because a shell that runs in the background ignores SIGINT. The Neovim then starts and runs until its own timeout, and two of the four come back after the runner deleted them: the process id file, which the file's shell writes again, and the state folder, which the Neovim makes again. A SIGTERM or SIGHUP sent to the whole process group stops that shell too, so nothing starts and nothing is left. The same holds for any signal that kills a runner that leads its own session on a terminal, for example one that a terminal emulator starts directly: its death hangs up the foreground process group of the terminal. It sets `VIMINIT`, `CLAUDECODE_MINIMAL_INIT`, `XDG_STATE_HOME` and `XDG_CONFIG_HOME` for that file's Neovims and the processes they start only, as a prefix of one command, and leaves `XDG_DATA_HOME` alone because plenary lives there. A run writes nothing under `~/.local/state/nvim`, and the developer's `ftplugin` and `after` files under `~/.config/nvim` do not run [run_integration_tests_individually.sh:temp_output, group_file, state_dir, config_dir]. +- `run_integration_tests_individually.sh` creates two `mktemp` files and two `mktemp -d` folders per test (the output, the file that holds the process id of the file's `timeout`, and the `XDG_STATE_HOME` and `XDG_CONFIG_HOME` that the file's two Neovims share) and deletes all four after the file ran, with `.status`, where the file's subshell writes the exit statuses of nvim and of `tee` when the file ends. When it gets SIGTERM, SIGHUP or SIGINT (Ctrl-C) while a file runs, it kills that `timeout`, then the process group of the file, deletes the four and dies by that signal at once, so a calling shell sees status 143, 129 or 130, and a shell loop around the runner stops after Ctrl-C. Killing the `timeout` first covers the moments before it has made its own group, when no Neovim runs yet. Both kills use KILL, so a Neovim that ignores SIGTERM stops too. Two windows of a few milliseconds are not covered. (1) While the runner makes its four items it has not stored their names yet: a signal then leaves the items it already made, never deletes them, and starts no Neovim. (2) Between the moment the runner stores the names and the moment the file's shell has written the process id, there is no `timeout` to kill. A SIGTERM or SIGHUP sent to the runner alone leaves that shell running, and so does any SIGINT (Ctrl-C in a shell, for one), because a shell that runs in the background ignores SIGINT. The Neovim then starts and runs until its own timeout, and two of the four come back after the runner deleted them: the process id file, which the file's shell writes again, and the state folder, which the Neovim makes again. A SIGTERM or SIGHUP sent to the whole process group stops that shell too, so nothing starts and nothing is left. The same holds when SIGTERM, SIGHUP, SIGINT or SIGUSR1 kills a runner that leads its own session on a terminal, for example one that a terminal emulator starts directly: its death hangs up the foreground process group of the terminal, so that shell dies too. SIGKILL gives the runner no time to clean up: a runner that leads such a session and gets it starts no Neovim, but all four items stay, and without a terminal the Neovim starts as well. Bash ignores SIGQUIT, so the file runs to its end. It sets `VIMINIT`, `CLAUDECODE_MINIMAL_INIT`, `XDG_STATE_HOME` and `XDG_CONFIG_HOME` for that file's Neovims and the processes they start only, as a prefix of one command, and leaves `XDG_DATA_HOME` alone because plenary lives there. A run writes nothing under `~/.local/state/nvim`, and the developer's `ftplugin` and `after` files under `~/.config/nvim` do not run [run_integration_tests_individually.sh:temp_output, group_file, state_dir, config_dir]. - `repro_issue_231.lua` deletes `/__issue_231_repro__.md` [repro_issue_231.lua:try_open_diff]. - `repro_issue_70.sh` and `repro_issue_277.sh` make a work dir with `mktemp -d` (`${TMPDIR:-/tmp}/issue70.XXXXXX` or `issue277.XXXXXX`) and an `att-home` folder in it before they check their tools. They install `trap cleanup EXIT` after that check, so an exit 3 for a missing tool leaves the work dir behind. The trap also runs on SIGTERM, SIGINT and SIGHUP. It destroys the `agent-tty` sessions and removes the work dir. agent-tty still leaves an empty socket folder `/tmp/agent-tty/<8 hex digits>` (it ignores `TMPDIR`) and an entry for the deleted home in `${XDG_STATE_HOME:-$HOME/.local/state}/agent-tty/homes.json`, one of each per run. The sessions inherit the caller's environment, so `CLAUDE_CONFIG_DIR` and the `XDG_*` variables reach the Neovim and the `claude` CLI that they run. `repro_issue_70.sh` launches the real `claude` CLI, and it also runs `claude --version` once outside agent-tty. Its cleanup runs `pkill -f "repro_issue_70_probe.lua $REPO_ROOT"`, which also kills the probe of another run in the same checkout [repro_issue_70.sh:WORK, cleanup] [repro_issue_277.sh:WORK, cleanup]. - `repro_issue_70_probe.lua` starts the real server with `auto_start = true`, so the plugin writes a real `PORT.lock` into the user's lock directory. Other Claude CLIs can see that lock file until `claudecode.stop()` removes it at the end. When the removal fails, `stop` only logs a warning, so a lock file that the removal failed to delete stays [repro_issue_70_probe.lua:claudecode.setup, claudecode.stop] [lua/claudecode/init.lua:M.stop]. diff --git a/scripts/run_integration_tests_individually.sh b/scripts/run_integration_tests_individually.sh index f2e7324af..48903e57c 100755 --- a/scripts/run_integration_tests_individually.sh +++ b/scripts/run_integration_tests_individually.sh @@ -61,12 +61,13 @@ run_test_file() { echo "" echo "Running: $basename" - local temp_output group_file state_dir config_dir + local temp_output group_file state_dir config_dir status_file temp_output=$(mktemp) group_file=$(mktemp) state_dir=$(mktemp -d) config_dir=$(mktemp -d) - CURRENT_TEMP=("$temp_output" "$group_file" "$state_dir" "$config_dir") + status_file=$temp_output.status + CURRENT_TEMP=("$temp_output" "$group_file" "$state_dir" "$config_dir" "$status_file") CURRENT_GROUP_FILE=$group_file # timeout runs nvim in a process group named after its own pid, which is this shell's pid until it execs timeout. @@ -79,10 +80,11 @@ run_test_file() { # folders, so the one started here takes -i NONE: two ShaDa writers made the first fail with E886 after a timeout. # The file runs in the background and the runner waits for it: `wait` ends at once when the INT trap runs. The # subshell has no pipefail, because with it bash prints a job report ("Killed", two lines) when the cleanup above - # kills `timeout`. It exits with the status of the first command. When `tee` fails, set -e ends the subshell with - # the status of `tee`, as pipefail did. + # kills `timeout`. It writes the exit statuses of nvim and of `tee` to the status file, so that the reason line can + # name the stage that failed, and exits with the status of nvim, or with that of `tee` when nvim exited 0. ( set +o pipefail + set +e VIMINIT='lua local f = vim.env.CLAUDECODE_MINIMAL_INIT if f then dofile(f) end' \ CLAUDECODE_MINIMAL_INIT="$PWD/tests/minimal_init.lua" \ XDG_STATE_HOME="$state_dir" XDG_CONFIG_HOME="$config_dir" \ @@ -90,7 +92,9 @@ run_test_file() { nvim --headless -i NONE -u tests/minimal_init.lua \ -c "lua require('plenary.test_harness').test_file('$test_file')" \ 2>&1 | tee "$temp_output" - exit "${PIPESTATUS[0]}" + pipe_status=("${PIPESTATUS[@]}") + echo "${pipe_status[0]} ${pipe_status[1]}" >"$status_file" + exit $((pipe_status[0] != 0 ? pipe_status[0] : pipe_status[1])) ) & if wait $!; then EXIT_CODE=0 @@ -98,6 +102,11 @@ run_test_file() { EXIT_CODE=$? fi CURRENT_GROUP_FILE="" + local tee_status=0 status_line + if status_line=$(cat "$status_file" 2>/dev/null) && [[ $status_line =~ ^([0-9]+)\ ([0-9]+)$ ]]; then + EXIT_CODE=${BASH_REMATCH[1]} + tee_status=${BASH_REMATCH[2]} + fi # Parse results from output local clean_output @@ -126,31 +135,41 @@ run_test_file() { TOTAL_ERRORS=$((TOTAL_ERRORS + error_count)) # Exit 124 is timeout's, 137 is its SIGKILL after 5 more seconds, and 143 is SIGTERM. + # An outside SIGTERM that reaches timeout gives 124 (uutils) or 143 (GNU), and the out-of-memory killer gives 137 like + # timeout's own KILL, so these lines do not claim that the limit was hit. if [[ $EXIT_CODE -eq 124 ]] || [[ $EXIT_CODE -eq 137 ]] || [[ $EXIT_CODE -eq 143 ]]; then if [[ $EXIT_CODE -eq 137 ]]; then - # The out-of-memory killer gives 137 too, so this line does not claim that the limit was hit. echo "TIMEOUT: $basename was stopped (exit 137, SIGKILL): nvim did not exit within $FILE_TIMEOUT," \ "or another process killed it" else - echo "TIMEOUT: $basename was stopped (exit $EXIT_CODE): nvim did not exit within $FILE_TIMEOUT" + echo "TIMEOUT: $basename was stopped (exit $EXIT_CODE): nvim did not exit within $FILE_TIMEOUT," \ + "or an outside signal stopped it" fi reason_printed=true - # nvim busy in a Lua loop ignores SIGTERM and outlives timeout. + fi + + # nvim busy in a Lua loop ignores SIGTERM and outlives timeout, and when the nvim started here died, the one that + # plenary started for the spec runs on. + if [[ $EXIT_CODE -ne 0 ]]; then kill -KILL -- "-$(cat "$group_file")" 2>/dev/null || true fi - # A file that failed with a non-zero exit status always gets one line with a reason: the one above, or this one. + # A file whose nvim exited non-zero gets a line with a reason: the no-summary line, the TIMEOUT line, or this one. A + # file with no summary that a timeout or a signal stopped gets the first two, as on main. if [[ $EXIT_CODE -ne 0 ]] && [[ $reason_printed == false ]]; then echo "FAILED: $basename: nvim exit $EXIT_CODE$(signal_note "$EXIT_CODE")" fi + if [[ $tee_status -ne 0 ]]; then + echo "FAILED: $basename: tee exit $tee_status$(signal_note "$tee_status")" + fi # Check if test failed - if [[ $failed_count -gt 0 ]] || [[ $error_count -gt 0 ]] || [[ $EXIT_CODE -ne 0 ]]; then + if [[ $failed_count -gt 0 ]] || [[ $error_count -gt 0 ]] || [[ $EXIT_CODE -ne 0 ]] || [[ $tee_status -ne 0 ]]; then FAILED_FILES+=("$basename") fi # Cleanup - rm -rf "$temp_output" "$group_file" "$state_dir" "$config_dir" + rm -rf "$temp_output" "$group_file" "$state_dir" "$config_dir" "$status_file" CURRENT_TEMP=() } diff --git a/tests/unit/integration_runner_spec.lua b/tests/unit/integration_runner_spec.lua index 4a50812d3..746e5027f 100644 --- a/tests/unit/integration_runner_spec.lua +++ b/tests/unit/integration_runner_spec.lua @@ -25,14 +25,14 @@ local function concat(...) end local STUB_TIMEOUT = concat({ "#!/bin/sh" }, SPEC_NAME, { - 'if [ -f "$STUB_DIR/$name.ignore_term" ]; then', + 'if [ -f "$STUB_DIR/$name.ignore_term" ] || [ -f "$STUB_DIR/$name.outside_term" ]; then', " pid=$$", ' (while kill -0 "$pid"; do', ' if [ -f "$STUB_DIR/$name.ready" ]; then kill -TERM "$pid"; break; fi', " sleep 0.05", " done) > /dev/null 2>&1 &", ' echo $! > "$STUB_DIR/$name.helper.pid"', - ' if [ "$1" = -k ]; then', + ' if [ -f "$STUB_DIR/$name.ignore_term" ] && [ "$1" = -k ]; then', " shift 2", ' set -- -k 1s "$@"', " fi", @@ -55,8 +55,8 @@ local STUB_NVIM = concat({ "#!/bin/sh" }, SPEC_NAME, { 'if [ -d "$XDG_STATE_HOME" ] && [ -d "$XDG_CONFIG_HOME" ]; then folders=present; fi', 'printf "%s\\t%s\\t%s\\t%s\\t%s\\t%s\\n" "${VIMINIT-unset}" "${CLAUDECODE_MINIMAL_INIT-unset}" \\', ' "${XDG_STATE_HOME-unset}" "${XDG_CONFIG_HOME-unset}" "$folders" "${XDG_DATA_HOME-unset}" >> "$STUB_DIR/env_seen"', - 'if [ -f "$STUB_DIR/$name.ignore_term" ]; then', - " trap '' TERM", + 'if [ -f "$STUB_DIR/$name.ignore_term" ] || [ -f "$STUB_DIR/$name.outside_term" ]; then', + ' if [ -f "$STUB_DIR/$name.ignore_term" ]; then trap "" TERM; fi', ' : > "$STUB_DIR/$name.ready"', " for _ in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20; do sleep 1; done", "fi", @@ -129,6 +129,18 @@ local function find_wide_digit_locale() end end +local tee_pipe = assert(io.popen("command -v tee 2> /dev/null")) +local real_tee = assert(tee_pipe:read("*l"), "needs tee") +tee_pipe:close() +-- Copies its input like tee, then exits with the status in $STUB_DIR/tee.exit when that file exists. +local STUB_TEE = { + "#!/bin/sh", + quote(real_tee) .. ' "$@"', + "status=$?", + 'if [ -f "$STUB_DIR/tee.exit" ]; then status=$(cat "$STUB_DIR/tee.exit"); fi', + 'exit "$status"', +} + local timeout_pipe = assert(io.popen("command -v timeout 2> /dev/null")) local real_timeout = timeout_pipe:read("*l") timeout_pipe:close() @@ -182,6 +194,7 @@ describe("scripts/run_integration_tests_individually.sh", function() shell("mkdir -p " .. quote(dir .. "/bin") .. " " .. quote(dir .. "/tests/integration")) write(dir .. "/bin/nvim", STUB_NVIM, true) write(dir .. "/bin/timeout", STUB_TIMEOUT, true) + write(dir .. "/bin/tee", STUB_TEE, true) end) after_each(function() @@ -202,14 +215,16 @@ describe("scripts/run_integration_tests_individually.sh", function() ---@field output string[] ---@field orphan boolean|nil ---@field ignore_term boolean|nil + ---@field outside_term boolean|nil An outside SIGTERM reaches timeout while the stub nvim sleeps, and nvim dies of it ---@field die_by string|nil A signal the stub nvim sends itself after its output, as `kill -` takes it ---@param specs ScriptSpec[] ---@param locale string|nil The LC_ALL to run the script under + ---@param tmp string|nil The TMPDIR to run the script under, a folder that exists ---@return string status ---@return string output ---@return string[] ran - local function run_script(specs, locale) + local function run_script(specs, locale, tmp) for _, spec in ipairs(specs) do write(dir .. "/tests/integration/" .. spec.name, {}) write(dir .. "/" .. spec.name .. ".out", spec.output) @@ -220,6 +235,9 @@ describe("scripts/run_integration_tests_individually.sh", function() if spec.ignore_term then write(dir .. "/" .. spec.name .. ".ignore_term", {}) end + if spec.outside_term then + write(dir .. "/" .. spec.name .. ".outside_term", {}) + end if spec.die_by then write(dir .. "/" .. spec.name .. ".die_by", { spec.die_by }) end @@ -227,7 +245,7 @@ describe("scripts/run_integration_tests_individually.sh", function() local command = "cd " .. quote(dir) .. " && TMPDIR=" - .. quote(dir:match("^(.*)/[^/]+$")) + .. quote(tmp or dir:match("^(.*)/[^/]+$")) .. " PATH=" .. quote(dir .. "/bin") .. ':"$PATH" STUB_DIR=' @@ -239,7 +257,7 @@ describe("scripts/run_integration_tests_individually.sh", function() local output = pipe:read("*a") pipe:close() for _, spec in ipairs(specs) do - local pid_file = spec.ignore_term and io.open(dir .. "/" .. spec.name .. ".helper.pid") + local pid_file = (spec.ignore_term or spec.outside_term) and io.open(dir .. "/" .. spec.name .. ".helper.pid") if pid_file then table.insert(helper_pids, pid_file:read("*l")) pid_file:close() @@ -655,6 +673,8 @@ describe("scripts/run_integration_tests_individually.sh", function() for _, case in ipairs({ { 139, ", SIGSEGV" }, { 134, ", SIGABRT" }, + { 130, ", SIGINT" }, + { 129, ", SIGHUP" }, { 1, "" }, }) do local exit_status, note = case[1], case[2] @@ -719,6 +739,75 @@ describe("scripts/run_integration_tests_individually.sh", function() assert.is_nil(output:find("FAILED:", 1, true), output) end) + it("names tee, not nvim, when tee fails after a clean summary", function() + write(dir .. "/tee.exit", { "3" }) + local status, output = run_script({ + { name = "tee_spec.lua", exit = 0, output = summary(2, 0, 0) }, + }) + + assert.are.equal("1", status, output) + assert.is_truthy(output:find("FAILED: tee_spec.lua: tee exit 3\n", 1, true), output) + assert.is_nil(output:find("nvim exit", 1, true), output) + assert.is_truthy(output:find("Failed test files:\n - tee_spec.lua", 1, true), output) + end) + + it("names both stages when nvim and tee fail", function() + write(dir .. "/tee.exit", { "141" }) + local status, output = run_script({ + { name = "both_spec.lua", exit = 1, output = summary(2, 0, 0) }, + }) + + assert.are.equal("1", status, output) + assert.is_truthy(output:find("FAILED: both_spec.lua: nvim exit 1\n", 1, true), output) + assert.is_truthy(output:find("FAILED: both_spec.lua: tee exit 141, SIGPIPE\n", 1, true), output) + assert.are.equal(2, occurrences(output, "FAILED:"), output) + end) + + it("words the TIMEOUT line so that it holds when an outside SIGTERM reached timeout", function() + local status, output = run_script({ + { name = "outside_spec.lua", exit = 0, output = hung_output("outside_spec.lua"), outside_term = true }, + }) + + local _, limit = timeout_limits() + assert.are.equal("1", status, output) + -- uutils timeout answers an outside SIGTERM with 124, GNU timeout with 143. + local code, text = output:match("TIMEOUT: outside_spec.lua was stopped %(exit (%d+)%): ([^\n]*)\n") + assert.is_truthy(code == "124" or code == "143", output) + assert.is_truthy(text, output) + assert.are.equal("nvim did not exit within " .. limit .. "s, or an outside signal stopped it", text) + assert.are.equal(1, occurrences(output, "TIMEOUT:"), output) + end) + + it("leaves no temp item behind after a file that failed, the exit status file included", function() + write(dir .. "/tee.exit", { "3" }) + shell("mkdir -p " .. quote(dir .. "/tmp")) + local status, output = run_script({ + { name = "a_spec.lua", exit = 0, output = summary(1, 0, 0) }, + { name = "b_spec.lua", exit = 139, output = summary(1, 0, 0) }, + }, nil, dir .. "/tmp") + + assert.are.equal("1", status, output) + assert.is_truthy(output:find("FAILED: b_spec.lua: nvim exit 139, SIGSEGV\n", 1, true), output) + local listing = assert(io.popen("ls -A " .. quote(dir .. "/tmp"))) + local left = listing:read("*a") + listing:close() + assert.are.equal("", left, "the runner left temp items behind") + end) + + it("kills what the file left running when its nvim died by a signal before any summary", function() + local status, output = run_script({ + { name = "segv_orphan_spec.lua", exit = 139, output = hung_output("segv_orphan_spec.lua"), orphan = true }, + }) + local pid_file = assert(io.open(dir .. "/segv_orphan_spec.lua.orphan.pid")) + orphan_pid = pid_file:read("*l") + pid_file:close() + + assert.are.equal("1", status, output) + assert.is_truthy(output:find("no plenary summary (nvim exit 139, SIGSEGV)", 1, true), output) + wait_until_stopped(orphan_pid) + assert.is_false(is_running(orphan_pid), "the process the file left running was killed") + end) + it("fails a file whose nvim exited 0 after passing tests without printing a summary", function() local status, output = run_script({ { name = "stopped_spec.lua", exit = 0, output = { row("Success", "first"), row("Success", "second") } }, From 51ed703b161b4a044482fc487d10d6cac3de171a Mon Sep 17 00:00:00 2001 From: Thomas Kosiewski Date: Thu, 12 Nov 2026 10:33:03 +0000 Subject: [PATCH 3/3] fix(runner): keep the exit status file in the state folder MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The file's subshell wrote its exit statuses next to the output file. A subshell that ended after the runner cleaned up (a signal stop) wrote the file again and left it in TMPDIR. The file now lives in the state folder: the runner's cleanup deletes it with the folder, and a late write finds the folder gone and fails without a message. A test stubs a `tee` that ends a second after the signal and fails when anything is left in TMPDIR or the runner prints "No such file". _Generated with xum • Model: anthropic:claude-sonnet-5-5 • Thinking: xhigh_ --- scripts/README.md | 2 +- scripts/run_integration_tests_individually.sh | 13 +++++++------ tests/unit/integration_runner_spec.lua | 16 +++++++++++++++- 3 files changed, 23 insertions(+), 8 deletions(-) diff --git a/scripts/README.md b/scripts/README.md index 5b8c95eff..26215c96e 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -84,7 +84,7 @@ The Lua repros run from the repository root with `nvim --headless -u NONE -l scr - CAUTION: Do not run the token-sending clients (`repro_issue_238.sh`, `repro_issue_248.sh`, and `repro_issue_277.sh`) on a shared machine. They pass the auth token on the `websocat` command line (`--header`), where other users can read it with `ps` [repro_issue_238.sh:WEBSOCAT] [repro_issue_248.sh:TOKEN] [repro_issue_277.sh:TOKEN]. - `repro_issue_238.sh`, `repro_issue_248.sh`, and `repro_issue_277.sh` connect to the plugin server on `127.0.0.1` [repro_issue_238.sh:PORT] [repro_issue_248.sh:PORT] [repro_issue_277.sh:PORT]. -- `run_integration_tests_individually.sh` creates two `mktemp` files and two `mktemp -d` folders per test (the output, the file that holds the process id of the file's `timeout`, and the `XDG_STATE_HOME` and `XDG_CONFIG_HOME` that the file's two Neovims share) and deletes all four after the file ran, with `.status`, where the file's subshell writes the exit statuses of nvim and of `tee` when the file ends. When it gets SIGTERM, SIGHUP or SIGINT (Ctrl-C) while a file runs, it kills that `timeout`, then the process group of the file, deletes the four and dies by that signal at once, so a calling shell sees status 143, 129 or 130, and a shell loop around the runner stops after Ctrl-C. Killing the `timeout` first covers the moments before it has made its own group, when no Neovim runs yet. Both kills use KILL, so a Neovim that ignores SIGTERM stops too. Two windows of a few milliseconds are not covered. (1) While the runner makes its four items it has not stored their names yet: a signal then leaves the items it already made, never deletes them, and starts no Neovim. (2) Between the moment the runner stores the names and the moment the file's shell has written the process id, there is no `timeout` to kill. A SIGTERM or SIGHUP sent to the runner alone leaves that shell running, and so does any SIGINT (Ctrl-C in a shell, for one), because a shell that runs in the background ignores SIGINT. The Neovim then starts and runs until its own timeout, and two of the four come back after the runner deleted them: the process id file, which the file's shell writes again, and the state folder, which the Neovim makes again. A SIGTERM or SIGHUP sent to the whole process group stops that shell too, so nothing starts and nothing is left. The same holds when SIGTERM, SIGHUP, SIGINT or SIGUSR1 kills a runner that leads its own session on a terminal, for example one that a terminal emulator starts directly: its death hangs up the foreground process group of the terminal, so that shell dies too. SIGKILL gives the runner no time to clean up: a runner that leads such a session and gets it starts no Neovim, but all four items stay, and without a terminal the Neovim starts as well. Bash ignores SIGQUIT, so the file runs to its end. It sets `VIMINIT`, `CLAUDECODE_MINIMAL_INIT`, `XDG_STATE_HOME` and `XDG_CONFIG_HOME` for that file's Neovims and the processes they start only, as a prefix of one command, and leaves `XDG_DATA_HOME` alone because plenary lives there. A run writes nothing under `~/.local/state/nvim`, and the developer's `ftplugin` and `after` files under `~/.config/nvim` do not run [run_integration_tests_individually.sh:temp_output, group_file, state_dir, config_dir]. +- `run_integration_tests_individually.sh` creates two `mktemp` files and two `mktemp -d` folders per test (the output, the file that holds the process id of the file's `timeout`, and the `XDG_STATE_HOME` and `XDG_CONFIG_HOME` that the file's two Neovims share) and deletes all four after the file ran. The file's subshell writes the exit statuses of nvim and of `tee` to a file in the state folder when the file ends, and a write that comes after the cleanup finds the folder gone and fails. When it gets SIGTERM, SIGHUP or SIGINT (Ctrl-C) while a file runs, it kills that `timeout`, then the process group of the file, deletes the four and dies by that signal at once, so a calling shell sees status 143, 129 or 130, and a shell loop around the runner stops after Ctrl-C. Killing the `timeout` first covers the moments before it has made its own group, when no Neovim runs yet. Both kills use KILL, so a Neovim that ignores SIGTERM stops too. Two windows of a few milliseconds are not covered. (1) While the runner makes its four items it has not stored their names yet: a signal then leaves the items it already made, never deletes them, and starts no Neovim. (2) Between the moment the runner stores the names and the moment the file's shell has written the process id, there is no `timeout` to kill. A SIGTERM or SIGHUP sent to the runner alone leaves that shell running, and so does any SIGINT (Ctrl-C in a shell, for one), because a shell that runs in the background ignores SIGINT. The Neovim then starts and runs until its own timeout, and two of the four come back after the runner deleted them: the process id file, which the file's shell writes again, and the state folder, which the Neovim makes again. A SIGTERM or SIGHUP sent to the whole process group stops that shell too, so nothing starts and nothing is left. The same holds when SIGTERM, SIGHUP, SIGINT or SIGUSR1 kills a runner that leads its own session on a terminal, for example one that a terminal emulator starts directly: its death hangs up the foreground process group of the terminal, so that shell dies too. SIGKILL gives the runner no time to clean up: a runner that leads such a session and gets it starts no Neovim, but all four items stay, and without a terminal the Neovim starts as well. Bash ignores SIGQUIT, so the file runs to its end. It sets `VIMINIT`, `CLAUDECODE_MINIMAL_INIT`, `XDG_STATE_HOME` and `XDG_CONFIG_HOME` for that file's Neovims and the processes they start only, as a prefix of one command, and leaves `XDG_DATA_HOME` alone because plenary lives there. A run writes nothing under `~/.local/state/nvim`, and the developer's `ftplugin` and `after` files under `~/.config/nvim` do not run [run_integration_tests_individually.sh:temp_output, group_file, state_dir, config_dir]. - `repro_issue_231.lua` deletes `/__issue_231_repro__.md` [repro_issue_231.lua:try_open_diff]. - `repro_issue_70.sh` and `repro_issue_277.sh` make a work dir with `mktemp -d` (`${TMPDIR:-/tmp}/issue70.XXXXXX` or `issue277.XXXXXX`) and an `att-home` folder in it before they check their tools. They install `trap cleanup EXIT` after that check, so an exit 3 for a missing tool leaves the work dir behind. The trap also runs on SIGTERM, SIGINT and SIGHUP. It destroys the `agent-tty` sessions and removes the work dir. agent-tty still leaves an empty socket folder `/tmp/agent-tty/<8 hex digits>` (it ignores `TMPDIR`) and an entry for the deleted home in `${XDG_STATE_HOME:-$HOME/.local/state}/agent-tty/homes.json`, one of each per run. The sessions inherit the caller's environment, so `CLAUDE_CONFIG_DIR` and the `XDG_*` variables reach the Neovim and the `claude` CLI that they run. `repro_issue_70.sh` launches the real `claude` CLI, and it also runs `claude --version` once outside agent-tty. Its cleanup runs `pkill -f "repro_issue_70_probe.lua $REPO_ROOT"`, which also kills the probe of another run in the same checkout [repro_issue_70.sh:WORK, cleanup] [repro_issue_277.sh:WORK, cleanup]. - `repro_issue_70_probe.lua` starts the real server with `auto_start = true`, so the plugin writes a real `PORT.lock` into the user's lock directory. Other Claude CLIs can see that lock file until `claudecode.stop()` removes it at the end. When the removal fails, `stop` only logs a warning, so a lock file that the removal failed to delete stays [repro_issue_70_probe.lua:claudecode.setup, claudecode.stop] [lua/claudecode/init.lua:M.stop]. diff --git a/scripts/run_integration_tests_individually.sh b/scripts/run_integration_tests_individually.sh index 48903e57c..9cc4d9f02 100755 --- a/scripts/run_integration_tests_individually.sh +++ b/scripts/run_integration_tests_individually.sh @@ -66,8 +66,8 @@ run_test_file() { group_file=$(mktemp) state_dir=$(mktemp -d) config_dir=$(mktemp -d) - status_file=$temp_output.status - CURRENT_TEMP=("$temp_output" "$group_file" "$state_dir" "$config_dir" "$status_file") + status_file=$state_dir/exit_statuses + CURRENT_TEMP=("$temp_output" "$group_file" "$state_dir" "$config_dir") CURRENT_GROUP_FILE=$group_file # timeout runs nvim in a process group named after its own pid, which is this shell's pid until it execs timeout. @@ -80,8 +80,9 @@ run_test_file() { # folders, so the one started here takes -i NONE: two ShaDa writers made the first fail with E886 after a timeout. # The file runs in the background and the runner waits for it: `wait` ends at once when the INT trap runs. The # subshell has no pipefail, because with it bash prints a job report ("Killed", two lines) when the cleanup above - # kills `timeout`. It writes the exit statuses of nvim and of `tee` to the status file, so that the reason line can - # name the stage that failed, and exits with the status of nvim, or with that of `tee` when nvim exited 0. + # kills `timeout`. It writes the exit statuses of nvim and of `tee` to a file in the state folder, so that the reason + # line can name the stage that failed, and exits with the status of nvim, or with that of `tee` when nvim exited 0. + # A subshell that ends after the cleanup above finds the folder gone, and its write fails. ( set +o pipefail set +e @@ -93,7 +94,7 @@ run_test_file() { -c "lua require('plenary.test_harness').test_file('$test_file')" \ 2>&1 | tee "$temp_output" pipe_status=("${PIPESTATUS[@]}") - echo "${pipe_status[0]} ${pipe_status[1]}" >"$status_file" + echo "${pipe_status[0]} ${pipe_status[1]}" 2>/dev/null >"$status_file" exit $((pipe_status[0] != 0 ? pipe_status[0] : pipe_status[1])) ) & if wait $!; then @@ -169,7 +170,7 @@ run_test_file() { fi # Cleanup - rm -rf "$temp_output" "$group_file" "$state_dir" "$config_dir" "$status_file" + rm -rf "$temp_output" "$group_file" "$state_dir" "$config_dir" CURRENT_TEMP=() } diff --git a/tests/unit/integration_runner_spec.lua b/tests/unit/integration_runner_spec.lua index 746e5027f..017614dce 100644 --- a/tests/unit/integration_runner_spec.lua +++ b/tests/unit/integration_runner_spec.lua @@ -132,12 +132,14 @@ end local tee_pipe = assert(io.popen("command -v tee 2> /dev/null")) local real_tee = assert(tee_pipe:read("*l"), "needs tee") tee_pipe:close() --- Copies its input like tee, then exits with the status in $STUB_DIR/tee.exit when that file exists. +-- Copies its input like tee, then exits with the status in $STUB_DIR/tee.exit when that file exists, after a second +-- when $STUB_DIR/tee.delay exists. local STUB_TEE = { "#!/bin/sh", quote(real_tee) .. ' "$@"', "status=$?", 'if [ -f "$STUB_DIR/tee.exit" ]; then status=$(cat "$STUB_DIR/tee.exit"); fi', + 'if [ -f "$STUB_DIR/tee.delay" ]; then sleep 1; fi', 'exit "$status"', } @@ -452,6 +454,7 @@ describe("scripts/run_integration_tests_individually.sh", function() ---@field window boolean|nil Send the signal when timeout has started but has not made its process group yet ---@field deaf_to_term boolean|nil The file's Neovim ignores SIGTERM ---@field caller_loop boolean|nil Run the runner twice in a calling shell and send the signal to their process group + ---@field late_tee boolean|nil The file's tee ends a second after the signal, long after the runner cleaned up ---@param signal string The signal to send ---@param run SignalRun @@ -469,6 +472,9 @@ describe("scripts/run_integration_tests_individually.sh", function() if run.deaf_to_term then write(dir .. "/slow_spec.lua.deaf_to_term", {}) end + if run.late_tee then + write(dir .. "/tee.delay", {}) + end local temp = dir .. "/tmp" shell("mkdir -p " .. quote(temp)) local runner_out = " > " .. quote(dir .. "/runner.out") .. " 2>&1" @@ -552,6 +558,14 @@ describe("scripts/run_integration_tests_individually.sh", function() assert.is_false(result.ended) end) + it("leaves no exit status file when the file's subshell ends after the runner cleaned up", function() + local result = signal_runner("TERM", { late_tee = true }) + + assert.are.equal("143", result.status) + assert.are.equal("", result.left, "the file's subshell wrote its exit statuses after the runner deleted its items") + assert.is_nil(result.runner_output:find("No such file", 1, true), result.runner_output) + end) + it("stops a file on SIGTERM that arrives after timeout started and before it made its process group", function() local result = signal_runner("TERM", { window = true })