Repository navigation
fix(runtime): fire send_after/start_timer timers from the timer thread - #10
Merged
Merged
Conversation
Each erlang:send_after/3 and erlang:start_timer/3 started an OS thread that slept until the timer fired or was cancelled. macOS allows 4,096 threads per process, so about 4,000 pending timers made the next thread spawn fail inside an extern "C" function and the program aborted. Pending message timers now wait in one registry ordered by deadline (timer/message.rs). The nera-timer thread, which already ends receive ... after waits, sleeps until the earlier of the two kinds of deadline and delivers due messages. Cancellation and delivery keep the per-timer lock, so cancel_timer results and the delivery-before-cancel guarantee are unchanged, and pending timers still keep deadlock detection from ending the program. Like a send, a delivery leaves the receiver's receive timer to the worker that runs it. On macOS this also moves message timers onto the kqueue wait that fires on time at utility and background priority. Adds the many_pending_timers fixture (50,000 pending timers), a start_timer case in scripts/test-native-timer.py, and includes the HTTP/2 machine check in the Cowlib benchmark again.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Each erlang:send_after/3 and erlang:start_timer/3 started an operating-system thread that slept until the timer fired. On macOS, about 4,000 pending timers hit the 4,096-thread limit and the program aborted.
What changed
Before and after
Before: A native program with 5,000 pending timers, or a loop of 3,000 Cowlib HTTP/2 exchanges, aborted with exit status 134.
After: Both finish as they do on BEAM, and 200,000 pending timers run on the same 13 threads as a program with none.
The change is inside the runtime and nothing on screen changes.
Checked
How it was checked
cargo build --release --locked -p nera-runtime, then -p nera, then the workspace
cargo clippy --workspace --all-targets --locked -- -D warnings: clean
cargo test --workspace --locked --no-fail-fast: all pass, including seven new or re-typed timer and scheduler tests
python3 scripts/test-allocator-config.py: pass
./target/release/nera test -j 1 --timeout 30: 467 passed, 2 skipped (fixtures that need external onecached modules)
The gate ran one main commit earlier; that commit changes only max/2 and min/2
New unit tests and the many_pending_timers fixture: ten consecutive passes each, with no retries
python3 scripts/test-native-timer.py on macOS arm64: 24 of 24 cases pass
Risk
Risk above low: Blast radius, medium — Timer thread and scheduler delivery: every native program that uses timers; Novelty, medium — New message-timer registry beside the existing receive-timer heap
Full risk assessment