Skip to content

test(shutdown): compare the packet demo's maps in any key order - #9

Merged
mremond merged 2 commits into
mainfrom
fm/task-3ec827d1
Nov 12, 2026
Merged

mremond merged 2 commits into
mainfrom
fm/task-3ec827d1

Conversation

@mremond

@mremond mremond commented Nov 12, 2026 •

Copy link
Copy Markdown
Member

Why

The opt-in shutdown probe failed on every run of main: its packet demo case expected map keys in source order, but Nera prints maps of up to 32 keys in atom-table order.

What changed

  • The probe reads the packet demo's two output lines as Erlang terms and compares each map as a set of entries, so key order no longer matters.
  • Every line, key and value must still be present, with the same spacing; a missing or cut line, a changed value, or a repeated or extra key fails.
  • Negative controls run when the probe starts, so the comparison cannot quietly become lenient.
  • The contributing guide says which part of the probe's output may vary.

Before and after

Before: On main the probe passed the first four programs, then stopped with "incomplete/unexpected output" on the packet demo, whose output was complete and correct.

After: The probe passes all ten cases (five programs, two scheduler settings) and still fails when the packet demo's output is incomplete or wrong.

Only an opt-in test script and the contributing guide change; nothing a person using Nera sees changes.

Checked

  • Main's probe on a release build of main, macOS arm64: Stops on the packet demo case: the output lists protocol before version
  • Changed probe on the same build, ten runs in a row: 10 of 10 runs pass all 10 cases; medians 0.3 to 2.5 ms; the packet demo printed the same text in all 600 samples
  • Changed probe with the demo altered to print one packet, a different ttl, or an extra map key: Each run fails on the packet demo case with incomplete/unexpected output
  • The probe's built-in negative controls: Pass: reordered keys match; nine broken outputs are rejected
How it was checked

cargo build --release --locked -p nera-runtime, then -p nera: passed

python3 scripts/test-native-shutdown.py at main's version: exit 1 on the handler case

python3 scripts/test-native-shutdown.py at this version: exit 0 on 11 runs (one, then ten in a row), 10 of 10 cases within the 5 ms budget each time

Same probe with demo/serverless/handler.erl temporarily altered three ways (not committed): exit 1 each time, on the handler case

HANDLER_OUTPUT is byte-identical to the previous expected string

python3 scripts/ci/changes.py on the changed files: every CI job runs, since the probe script has no rule of its own

Risk

Risk above low: Novelty, medium — A small new reader for printed Erlang maps, tuples, atoms and integers

Full risk assessment
Dimension Level Why
Blast radius low One opt-in test script and one paragraph of the contributing guide
Behavior low Fixes a probe that failed on every run; no Nera behavior changes
Undo low Revert and done
Evidence low Probe run end to end before and after, plus three altered-demo runs that fail
Sensitivity low None
Novelty medium A small new reader for printed Erlang maps, tuples, atoms and integers
Dependencies low Standard library only

The shutdown probe compared the packet demo's output with a fixed string
that lists map keys in source order. Nera prints maps of up to 32 keys in
atom-table order, so the handler case failed on every run even though the
output was complete and correct.

The probe now reads those two lines as Erlang terms and compares maps as
sets of entries. Every line, key and value must still be there, spacing
included; a missing or cut line, a changed value or a repeated key fails.
Negative controls run at start-up so the comparison cannot quietly become
lenient.
@mremond mremond added this to the 0.1.1 milestone Nov 12, 2026
@mremond
mremond merged commit 86fa073 into main Nov 12, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant