Skip to content

Resolve parameter nodes at instant on first read - #570

Open
MaxGhenis wants to merge 10 commits into
masterfrom
lazy-parameter-node-at-instant
Open

MaxGhenis wants to merge 10 commits into
masterfrom
lazy-parameter-node-at-instant

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Taken over by Anthony Volk (anth-volk) on 2026-11-17 as part of the work on report runtimes (see the comment above). Max's 4 commits (bcf2094, 98bb55e, c58dede, 31469a4) are unchanged; current master (175e7ee) is merged in and the fixes below are added on top. His original measurements are kept under "Max's measurements".

Problem

ParameterNodeAtInstant.__init__ built every descendant up front. The first time a tax-benefit system was asked for its parameters at a date, it built that date's whole tree of nodes at instant, and ParameterNode._at_instant_cache (plus TaxBenefitSystem._parameters_at_instant_cache) kept it for the system's lifetime. Every reform system, whose caches start empty, paid again.

Change

Max's commits:

  • A node at instant resolves a child the first time it's read (__getattr__, __getitem__) and keeps it. It resolves all of its direct children only when something needs all of them: _children, iteration, repr, dir, vectorial indexing. A child read as an attribute becomes a plain instance attribute, so later reads cost what they did.
  • Which children a node has, and in what order, is fixed when it's built. A child resolves once, under one module-level reentrant lock that later reads don't take. Children named like the node's own attributes shadow them as before, and pickles of up-front nodes still load.

Added on top, each with failing tests first:

  • ParameterNode.add_child clears the cached nodes at instant of the node and of every node above it, which were built without the new child. It walks up without recursing and stops at a node it has already visited. While a node loads, a child named parent (policyengine-us has gov.hhs.medicaid.eligibility.categories.parent) replaces the node's parent attribute and points back at it. The walk clears before the child is set as an attribute and only clears a real dict, so a child named _at_instant_cache is fine too.
  • Parent links for scale brackets and homogenized nodes. ParameterScale sets each loaded bracket's parent, as clone already did (the same line as Link scale brackets and homogenized nodes to their parents #660). homogenize_parameter_structures links the nodes it puts in place of missing breakdown values (policyengine-us has 9,811). Without these links, Parameter.update below them cleared caches only up to them. On master that left stale values; with lazy nodes, whether a later read saw the update depended on what had been read before. The links also let mark_as_modified reach the scale and the nodes above it, as in a cloned tree, which Simulation.check_macro_cache reads.
  • Copies and pickles are snapshots. A deep copy or pickle of a node at instant resolves every child first. It holds the values the up-front build held and no parameter nodes. Before, a partly read node carried its unread parameter nodes, and through their parents the whole tree: a policyengine-us CTC node pickled to 128 MB, against 2.4 kB on master. A copied cached node keeps the values it was copied with, as on master.
  • Copying and pickling while other threads read. Reads now add to a node's __dict__, its lazy state and the _at_instant_cache of the parameter nodes below it. Copying or pickling those dictionaries directly raised RuntimeError: dictionary changed size during iteration if a child was resolved meanwhile.
    • A node at instant's copy state is a copy of its __dict__.
    • A parameter node's cache is an _AtInstantCache: a dict whose copy or pickle starts from a one-step copy of its items and still keeps it one object for nodes that share it.
    • _get_at_instant replaces a plain dict cache set from outside core (policyengine-us's copy_parameter_nodes sets one) before adding to it.
    • The lazy state now pickles under protocols 0 and 1. That fixes the 8 tests from Let vectorial parameter nodes be copied and pickled #584 that failed in every Test job of this PR's earlier CI run.
  • The up-front build where callers relied on it.
    • A node at an instant that isn't a date, or a date that doesn't exist (parameters(None) reads "None"; "2015-02-29"), is built up front, so it fails, or succeeds, as before. Simulation._calculate reads parameters(period).gov.abolitions[...] with period None and ignores errors; a lazy node there would have read the latest values.
    • Any subclass of ParameterNodeAtInstant gets the up-front build, so its add_child, __getattr__ and __getitem__ see every child as before.
    • A node's children are the ones node.children.items() gives, as the up-front build iterated them.
  • Forked processes get a new resolution lock (os.register_at_fork). A process forked while another thread held the lock could never resolve a child.

Same values as before

  • For any instant and any order of reads, every read returns what the up-front build held, and a node read in full equals it, child order included. Max's Hypothesis property test checks this against a reference that walks the ParameterNode tree directly.
  • Nodes requested after Parameter.update, add_child or a cache reset read the change, as before. With the parent links above, that now includes changes below scale brackets and homogenized nodes, where master kept serving stale values.
  • Intended differences (in the class docstring):
    • A node at instant held across an in-place change to its parameters reads the new value for a child it hasn't read yet; the up-front node held every value from construction. In Core, the long-lived holder is TaxBenefitSystem._parameters_at_instant_cache, which nothing clears after Parameter.update or Simulation.apply_reform (Define how Core's parameter caches are owned, invalidated, copied and shared #621, problem 1). Formulas read through the root's _at_instant_cache, which those clear. Removing this would take either the up-front build or Define how Core's parameter caches are owned, invalidated, copied and shared #621's rule that trees don't change after construction.
    • A child that fails to build at a date (policyengine-uk's NI scale refuses some reforms) fails when it's read, not when the node is built, so the node's other children can still be read. A deep copy or pickle of a tree that holds such a node builds it and fails then.
    • A pickle of a node at instant holds this version's lazy state, which earlier versions of core can't load. A pickled cache is a plain dict.

policyengine-us's spm.py

policyengine_us/spm.py (main e283430b2e) handles these caches directly:

  • it gives a traced request a shallow copy of the root with its own _at_instant_cache;
  • it shares one system's _parameters_at_instant_cache between policies;
  • it path-copies parameter nodes with copy_parameter_nodes;
  • it resets both caches when it detaches a tree.

All of this works unchanged. A lazy node resolves children through the shared child nodes' caches, so it holds the same objects the up-front build did. test_parameter_node_at_instant_systems.py follows each pattern in Core. A script against policyengine-us main read identical values and parameter-tree hashes on master and this branch (20 of 20): isolate_parameter_tracing through a traced simulation, a branch sharing a policy and then a reform that detaches it, pin_spm_policy, clone_spm_system, copy_parameter_nodes with an update, and share_spm_policy.

Interaction with #660

#660 also changes ParameterNode.add_child: it moves the body into _link_child and adds replace_child. Whichever PR merges second has one textual conflict there. To resolve it, keep #660's _check_child_type(child) and self._link_child(name, child), and call _clear_at_instant_caches_upward(self) just before _link_child. The bracket.parent = self line is the same in both and merges cleanly, and so does the homogenization change (this PR links parents after homogenizing; #660 uses replace_child while homogenizing). This branch's tests all pass on the merge of the two.

With #660, children named like node attributes aren't set as attributes, so the loop the add_child walk guards against no longer happens, and its parent links make this PR's homogenization pass find nothing to link.

Relation to #621

#621 proposes rules for Core's parameter caches and suggested lazy nodes wait for them. This PR doesn't wait. It keeps every value a node requested after a change reads, and it handles the three problems #621 lists for this PR's earlier version:

  • Mixed stale and fresh values after a missing invalidation. add_child clears now, and brackets and homogenized nodes have parents. The held-node case above is what remains.
  • Copying a tree while another thread read it raised RuntimeError. Fixed, with deterministic tests.
  • Order-dependent copy logic and a per-thread pickle counter. Not used: a copy resolves every child first.

The rest of #621 (freezing trees, removing the system-level cache, not copying caches) is unchanged and still applies.

Measurements

Re-measured on 2026-11-17 against master 175e7ee. Setup:

  • policyengine-us main e283430b2e (2.144.4) on Python 3.13, with its default dataset (populace_us_2024.h5);
  • subsample(n=2862, seed=0), which gives 2,490 households;
  • household_net_income for 2026;
  • one fresh process per run, running two phases:
    • First calculation: Microsimulation(), the subsample, then the calculation.
    • Reform system with empty caches: then, in the same process, Microsimulation(reform={"gov.irs.deductions.standard.amount.SINGLE": {"2026-01-01.2100-12-31": 20000}}) (policyengine-us builds a new system), the same subsample and the same calculation.
master this branch
Nodes at instant built, first calculation 665,933 121,349
Nodes at instant built, reform system 699,744 122,008
Nodes at instant alive after both phases (two systems) 1,399,464 243,997
Calculation CPU time, reform system, median of 5 31.5 s 21.3 s
Calculation CPU time, first calculation, median of 5 27.3 s 28.2 s

Outputs are identical. Every stored array of both simulations and their branches matches master: 428,303 arrays, hashed over dtype, shape and bytes. So do the weighted totals.

Times. Runs were interleaved (master, branch, master, ...) on a shared machine with a load average of about 20 on 10 cores, so single runs vary by several seconds.

  • Reform system: the branch was faster in all 5 pairs, by 6.0 to 12.5 s of CPU time.
  • First calculation: no difference showed. The paired differences ran from 8.3 s slower to 3.3 s faster, median 0.6 s slower.
  • Wall times, from an earlier set of 3 pairs, agree: reform system 28.0 s against 21.5 s, first calculation 31.1 s against 30.7 s.
  • On master, the reform system's calculation runs with the first system's trees still alive, which a fresh process's first calculation doesn't have; that difference may explain why only the reform system got faster here. This wasn't measured separately.

Memory. Peak resident memory couldn't be compared on this machine: the medians were 6.94 GB against 5.54 GB in one set of runs, and 5.43 GB against 5.44 GB in the next.

Node objects were counted by wrapping ParameterNodeAtInstant.__init__ in a separate run, and live objects with gc after each phase.

Max's measurements (2026-10-02)

Measured on policyengine-us (132,396 parameter descendants) on master b78b0ba:

  • each instant cost about 34,000 node objects, 16 to 33 MB, and 4 to 5 s on a loaded host, on every system;
  • a household on a reform system added 349 MB, most of it the reform system's own at-instant trees.

A YAML run touches every year it tests and every month of a monthly variable, so the shared system and each reform system all pay this many times over. It was the third cause of the 118 GB policyengine-core test runs on 2026-10-02, next to the test-runner causes fixed in #569; the two PRs are independent. Combined with #569 on the two policyengine-us CI batches that failed on 16 GB runners, peak memory fell from 16.3 GB to 7.0 GB (NY/OH/OK batch) and from 17.1 GB to 6.4 GB (USDA batch): #569 (comment).

For downstream packages

  • policyengine-uk. Two of its tests on main rely on the up-front build, and will fail once it takes a core with this change. Simulation outputs are not affected.
    • tests/code_health/test_labour_supply_parameters_are_read.py records parameter reads by patching ParameterNodeAtInstant.__getattribute__. A first read of a child now goes through __getattr__, and object.__getattribute__(node, "_children") raises until the node is read in full. So it misses reads, and the test reports four labour supply parameters as unread.
    • test_uprating_inputs_reach_data.py::test_a_failing_probe_leaves_the_system_and_its_caches_alone expects reading the root at an instant to fill descendants' caches.
  • policyengine-household-api. prewarm_parameter_caches calls get_parameters_at_instant for many instants so the memory snapshot holds built trees. It now builds only each root; the first requests resolve what they read.

Tests

Added or changed, run file by file:

  • tests/core/parameters/test_parameter_node_at_instant_lazy.py
  • test_parameter_node_at_instant_property.py
  • test_parameter_node_add_child_caches.py
  • test_parameter_node_at_instant_copies.py
  • test_parameter_node_at_instant_systems.py

All 235 pass, and they also pass on a merge with #660. The rest of the suite wasn't run locally. Five rounds of independent adversarial review: the first four found bugs, each fixed with a failing test first; the fifth found none in Core.

Credit: the lazy resolution, its thread safety and its property tests are Max Ghenis's work.

🤖 Generated with Claude Code

@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Measurements of this PR combined with #569 on the two policyengine-us CI batches that are failing on 16 GB runners: see #569 (comment). Together they take the NY/OH/OK batch from 16.3 GB (3.32.11) to 7.0 GB and the USDA batch from 17.1 GB to 6.4 GB peak footprint.

@anth-volk

Copy link
Copy Markdown
Collaborator

Rebased onto master (b909d5a6) and force-pushed: de47cde0 → 089761c5. Beyond the rebase:

Details are under "October 14 rebase" in the description.

anth-volk added a commit that referenced this pull request Oct 14, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk added a commit that referenced this pull request Oct 14, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk added a commit that referenced this pull request Oct 14, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk added a commit that referenced this pull request Oct 14, 2026
#570 now provides the walk, so modify_parameters uses it instead of its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk added a commit that referenced this pull request Oct 15, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MaxGhenis and others added 4 commits October 16, 2026 00:20
Building every descendant in the constructor made each instant a system is
asked about cost a copy of the whole parameter tree, kept for the system's
lifetime. For policyengine-us that is about 34,000 node objects, 16-33 MB and
several seconds per instant, per tax-benefit system.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-coherent

Review of 404a3a3 found five regressions against the up-front build:
- concurrent first reads could raise KeyError/AttributeError or resolve a
  child twice: first resolution now runs under a module RLock;
- children named like the new state or helpers (_node, _resolved, _resolve,
  ...) read as internal objects: state is name-mangled, helpers are module
  functions, and children named like the node's own attributes resolve at
  construction, as before;
- a shallow copy and its original could resolve a scale twice: both now read
  one grow-only resolved map;
- a child with no value at the instant was re-resolved on every read: its
  absence is cached;
- deepcopy/pickle of a traced tree's raw node recursed through
  TracingParameterNodeAtInstant.__getattr__, which now raises AttributeError
  for its own unset fields.
A node also drops its parameter node once every child has been read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ases

Round-2 review of a90b524 found reads that still differed from the up-front
build. A node now keeps all of its own state in one _LazyChildren object under
a single reserved name (a parameter of that name is rejected):
- which children a node has, and their order, are fixed at construction
  (a child deleted from the tree later keeps its place);
- completion counts only those children, so add_child extras can no longer
  send the first read into recursive completion;
- resolution results and add_child entries are kept apart: items are shared
  by shallow copies, attributes belong to each copy, as before;
- a copy installs what it reads as attributes (no lock on warm reads);
- children named like attributes of the node's runtime type (dunders and
  subclass methods included) resolve at construction; a subclass overriding
  add_child gets the up-front build through it;
- a reentrant read keeps the first recorded child;
- nodes pickled by the up-front build load (__setstate__).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk

Copy link
Copy Markdown
Collaborator

I've dropped the commits I added to this branch and restored Max's description.

  • Dropped:
    • 089761c5, which let lazy nodes pickle under protocols 0 and 1;
    • 62d63079 and b26aa0a7, cache-consistency fixes;
    • 5719353e, a merge of master;
    • four review fixes that were never pushed.
  • Now on the branch: only Max's 4 commits, rebased onto master (4adfea70) and force-pushed: 5719353e → 31469a4d. The October 14 rebase's one change to Max's code remains: the _OWN_ATTRIBUTES guard in the tracing wrapper is gone, because master's _wrapped covers that case.
  • Expected CI failures: without 089761c5, four of master's copy-and-pickle tests from Let vectorial parameter nodes be copied and pickled #584 fail on this branch, the protocol 0 and 1 cases of:
    • test_tracing_wrapper_of_a_slotted_node_can_be_pickled;
    • test_copied_parameter_tree_keeps_its_cached_vectorial_node.

Review kept finding edge cases in how Core's parameter caches are filled, invalidated, copied and shared. Some predate this PR; others come from nodes changing their own state as they're read. #621 proposes rules for those caches, which this PR should wait for.

anth-volk and others added 2 commits November 17, 2026 13:47
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Finishes the lazy nodes at instant:

- ParameterNode.add_child clears the nodes at instant cached by the node
  and every node above it, which were built without the new child. It
  walks up without recursing and stops at a node it has seen: while a node
  loads, a child named "parent" (policyengine-us's
  gov.hhs.medicaid.eligibility.categories.parent) replaces the node's
  parent attribute and points back at it.
- Copies and pickles take copies of the dictionaries that reads change:
  a node at instant's __dict__, its shared lazy state and a parameter
  node's _at_instant_cache. Copying a tree while another thread resolved a
  child raised "dictionary changed size during iteration". The lazy state
  now also pickles under protocols 0 and 1, which master's copy and pickle
  tests need. A shallow copy of a parameter node still shares its cache,
  and slotted subclasses keep their slots.
- A node at an instant that is not a date, such as parameters(None), is
  built up front, so it fails, or succeeds, as the up-front build did.
  Simulation._calculate reads parameters(None).gov.abolitions for a
  variable calculated without a period and ignores the error.
- A forked process gets a new resolution lock, so a fork while another
  thread resolves a child doesn't leave the lock held for ever.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk

Copy link
Copy Markdown
Collaborator

@MaxGhenis, I'm taking this PR over as part of the work on report runtimes, and I'll keep working on this branch. Your 4 commits stay as they are, with your authorship: I'll merge current master in and add commits on top, so there won't be another force-push. I'll also update the description as the work goes, keeping your measurements.

Thanks for the groundwork here!

Fixes from the first review round:

- A deep copy or pickle of a node at instant resolves every child first,
  so it is a snapshot that holds no parameter nodes, as when every child
  was built up front. Before, a partly read node carried its unread
  parameter nodes, and through their parents the whole tree: 128 MB and
  22 s to deep-copy a policyengine-us CTC node, against about nothing on
  master. A copied cached node keeps the values it was copied with, as on
  master.
- The lazy state is copied as it is once complete, so a copy's
  _children and its state stay one mapping and add_child on the copy is
  listed.
- add_child on a fully read node no longer changes what attribute reads
  return on its other copies (the resolved children and the items were
  one dictionary). Items are read from the complete mapping, so direct
  writes to _children are items, as before.
- A parameter node's at-instant cache is an _AtInstantCache, a dict that
  copies and pickles from a one-step copy of its items. Copies stay safe
  while other threads read, and parameter nodes sharing one cache still
  share its copy. ParameterNode's own __getstate__ and __copy__ are gone.
- ParameterScale sets each loaded bracket's parent, as clone already did
  and as #660 does. Parameter.update and add_child below a bracket now
  clear the caches above the scale, so a reform of a bracket reads the
  same whatever was calculated before it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk and others added 3 commits November 17, 2026 15:27
Fixes from the second review round:

- homogenize_parameter_structures links the nodes it puts in place of
  missing breakdown values to their parent node. They were set straight
  into children with no parent, so an update below one cleared caches
  only up to it, and whether a later read saw the update depended on
  what had been read before (policyengine-us has 9,811 such nodes).
- A subclass of ParameterNodeAtInstant that overrides add_child gets
  the up-front build as before: every child added into _children, which
  is also the mapping items are read from. An add_child that wrote
  _children directly lost its items.
- A node at a date that does not exist ("2015-02-29", "0000-01-01") is
  built up front, so it fails as before. periods.instant reads these;
  str(Instant), which a scale's build calls, refuses them.
- A node's children are the ones node.children.items() gives, as the
  up-front build iterated them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes from the third review round:

- Any subclass of ParameterNodeAtInstant gets the up-front build, not
  only one that overrides add_child. A subclass whose __getattr__ or
  __getitem__ answers names that are not children returned its fallback
  for real children not read yet.
- ParameterNode._get_at_instant replaces a plain dict cache, with its
  entries, by an _AtInstantCache before adding to it. policyengine-us's
  copy_parameter_nodes sets plain dict caches, as do trees pickled by an
  earlier core, and copying or pickling such a tree while another thread
  resolved a child raised "dictionary changed size during iteration".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes from the fourth review round:

- ParameterNode._get_at_instant stores the node it built into the cache
  the node holds after building it, as before. If code run while the
  node was built replaced the cache (policyengine-us's
  isolate_parameter_tracing does, to drop an earlier request's nodes),
  the old entries came back.
- A complete node's resolved children are no longer the same dictionary
  as its items, so a write into _children changes items only, as when
  every child was built up front, and attributes keep what the
  parameters give.
- A pickled _AtInstantCache is a plain dict, so trees pickled by this
  version load in earlier versions of core when they hold no nodes at
  instant.
- The class docstring says that a child that fails to build fails when
  read, and that a pickle of a node at instant holds this version's lazy
  state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk
anth-volk marked this pull request as ready for review November 17, 2026 15:24

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants