Skip to content

fix(runtime): fire send_after/start_timer timers from the timer thread - #10

Merged
mremond merged 2 commits into
mainfrom
fm/task-7a699573
Nov 12, 2026
Merged

mremond merged 2 commits into
mainfrom
fm/task-7a699573

Conversation

@mremond

@mremond mremond commented Nov 12, 2026

Copy link
Copy Markdown
Member

Why

Each erlang:send_after/3 and erlang:start_timer/3 started an operating-system thread that slept until the timer fired. On macOS, about 4,000 pending timers hit the 4,096-thread limit and the program aborted.

What changed

  • Pending send_after and start_timer timers wait in one registry ordered by deadline. The runtime's existing timer thread delivers them, so a pending timer costs memory instead of a thread.
  • cancel_timer results are unchanged. A cancel that loses to a delivery still returns only after the message is in the mailbox.
  • On macOS these messages now arrive on time at utility and background priority.
  • Like a send, a timer delivery leaves the receiver's receive timeout to the worker that runs it.
  • Adds a 50,000-timer fixture and a start_timer case to the timer probe. The HTTP/2 check is back in the Cowlib benchmark.

Before and after

Before: A native program with 5,000 pending timers, or a loop of 3,000 Cowlib HTTP/2 exchanges, aborted with exit status 134.

After: Both finish as they do on BEAM, and 200,000 pending timers run on the same 13 threads as a program with none.

The change is inside the runtime and nothing on screen changes.

Checked

  • 5,000 pending start_timer calls, then cancel_timer on each, on macOS arm64: Before: aborted with 'failed to spawn thread', exit status 134. After: exit status 0, with the same output as OTP 29.
  • OS threads while timers are pending: Before: 1,013 threads at 1,000 timers and 3,013 at 3,000, then an abort at 5,000. After: 13 threads at 0, 5,000, 50,000 and 200,000 timers.
  • Cowlib HTTP/2 machine loop (four 5 s timers per exchange): Before: 280 us per exchange at 1,000 exchanges, and an abort at 3,000. After: 1,000, 3,000 and 20,000 exchanges all finish, at 63 to 68 us each.
  • Native timer probe: a 37 ms start_timer at normal, utility and background priority: Before: median 64 ms late at utility and 82 ms late at background. After: median 0.1 to 0.2 ms late everywhere, and all 24 probe cases pass.
  • Memory with 200,000 pending timers: Native peak footprint is 168 MB, against 3.6 MB with none. OTP 29 goes from 67 MB to 156 MB.
  • many_pending_timers fixture on BEAM and on the old runtime: It passes on OTP 29 and crashes the old native runtime.
How it was checked

cargo build --release --locked -p nera-runtime, then -p nera, then the workspace

cargo clippy --workspace --all-targets --locked -- -D warnings: clean

cargo test --workspace --locked --no-fail-fast: all pass, including seven new or re-typed timer and scheduler tests

python3 scripts/test-allocator-config.py: pass

./target/release/nera test -j 1 --timeout 30: 467 passed, 2 skipped (fixtures that need external onecached modules)

The gate ran one main commit earlier; that commit changes only max/2 and min/2

New unit tests and the many_pending_timers fixture: ten consecutive passes each, with no retries

python3 scripts/test-native-timer.py on macOS arm64: 24 of 24 cases pass

Risk

Risk above low: Blast radius, medium — Timer thread and scheduler delivery: every native program that uses timers; Novelty, medium — New message-timer registry beside the existing receive-timer heap

Full risk assessment
Dimension Level Why
Blast radius medium Timer thread and scheduler delivery: every native program that uses timers
Behavior low Defect corrected; timer results, order and cancel semantics are unchanged
Undo low Revert and done
Evidence low Reproduced and shown fixed end to end on macOS; Linux runs in CI
Sensitivity low No saved data, sign-ins, keys or releases
Novelty medium New message-timer registry beside the existing receive-timer heap
Dependencies low No new or updated packages

Each erlang:send_after/3 and erlang:start_timer/3 started an OS thread
that slept until the timer fired or was cancelled. macOS allows 4,096
threads per process, so about 4,000 pending timers made the next thread
spawn fail inside an extern "C" function and the program aborted.

Pending message timers now wait in one registry ordered by deadline
(timer/message.rs). The nera-timer thread, which already ends
receive ... after waits, sleeps until the earlier of the two kinds of
deadline and delivers due messages. Cancellation and delivery keep the
per-timer lock, so cancel_timer results and the delivery-before-cancel
guarantee are unchanged, and pending timers still keep deadlock
detection from ending the program. Like a send, a delivery leaves the
receiver's receive timer to the worker that runs it.

On macOS this also moves message timers onto the kqueue wait that fires
on time at utility and background priority.

Adds the many_pending_timers fixture (50,000 pending timers), a
start_timer case in scripts/test-native-timer.py, and includes the
HTTP/2 machine check in the Cowlib benchmark again.
@mremond mremond added this to the 0.1.1 milestone Nov 12, 2026
@mremond
mremond merged commit 788b1af into main Nov 12, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant