Repository navigation
Conversation
|
Measurements of this PR combined with #569 on the two policyengine-us CI batches that are failing on 16 GB runners: see #569 (comment). Together they take the NY/OH/OK batch from 16.3 GB (3.32.11) to 7.0 GB and the USDA batch from 17.1 GB to 6.4 GB peak footprint. |
de47cde to
089761c
Compare
|
Rebased onto
Details are under "October 14 rebase" in the description. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#570 now provides the walk, so modify_parameters uses it instead of its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Building every descendant in the constructor made each instant a system is asked about cost a copy of the whole parameter tree, kept for the system's lifetime. For policyengine-us that is about 34,000 node objects, 16-33 MB and several seconds per instant, per tax-benefit system. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…-coherent Review of 404a3a3 found five regressions against the up-front build: - concurrent first reads could raise KeyError/AttributeError or resolve a child twice: first resolution now runs under a module RLock; - children named like the new state or helpers (_node, _resolved, _resolve, ...) read as internal objects: state is name-mangled, helpers are module functions, and children named like the node's own attributes resolve at construction, as before; - a shallow copy and its original could resolve a scale twice: both now read one grow-only resolved map; - a child with no value at the instant was re-resolved on every read: its absence is cached; - deepcopy/pickle of a traced tree's raw node recursed through TracingParameterNodeAtInstant.__getattr__, which now raises AttributeError for its own unset fields. A node also drops its parameter node once every child has been read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ases Round-2 review of a90b524 found reads that still differed from the up-front build. A node now keeps all of its own state in one _LazyChildren object under a single reserved name (a parameter of that name is rejected): - which children a node has, and their order, are fixed at construction (a child deleted from the tree later keeps its place); - completion counts only those children, so add_child extras can no longer send the first read into recursive completion; - resolution results and add_child entries are kept apart: items are shared by shallow copies, attributes belong to each copy, as before; - a copy installs what it reads as attributes (no lock on warm reads); - children named like attributes of the node's runtime type (dunders and subclass methods included) resolve at construction; a subclass overriding add_child gets the up-front build through it; - a reentrant read keeps the first recorded child; - nodes pickled by the up-front build load (__setstate__). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5719353 to
31469a4
Compare
|
I've dropped the commits I added to this branch and restored Max's description.
Review kept finding edge cases in how Core's parameter caches are filled, invalidated, copied and shared. Some predate this PR; others come from nodes changing their own state as they're read. #621 proposes rules for those caches, which this PR should wait for. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Finishes the lazy nodes at instant: - ParameterNode.add_child clears the nodes at instant cached by the node and every node above it, which were built without the new child. It walks up without recursing and stops at a node it has seen: while a node loads, a child named "parent" (policyengine-us's gov.hhs.medicaid.eligibility.categories.parent) replaces the node's parent attribute and points back at it. - Copies and pickles take copies of the dictionaries that reads change: a node at instant's __dict__, its shared lazy state and a parameter node's _at_instant_cache. Copying a tree while another thread resolved a child raised "dictionary changed size during iteration". The lazy state now also pickles under protocols 0 and 1, which master's copy and pickle tests need. A shallow copy of a parameter node still shares its cache, and slotted subclasses keep their slots. - A node at an instant that is not a date, such as parameters(None), is built up front, so it fails, or succeeds, as the up-front build did. Simulation._calculate reads parameters(None).gov.abolitions for a variable calculated without a period and ignores the error. - A forked process gets a new resolution lock, so a fork while another thread resolves a child doesn't leave the lock held for ever. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@MaxGhenis, I'm taking this PR over as part of the work on report runtimes, and I'll keep working on this branch. Your 4 commits stay as they are, with your authorship: I'll merge current Thanks for the groundwork here! |
Fixes from the first review round: - A deep copy or pickle of a node at instant resolves every child first, so it is a snapshot that holds no parameter nodes, as when every child was built up front. Before, a partly read node carried its unread parameter nodes, and through their parents the whole tree: 128 MB and 22 s to deep-copy a policyengine-us CTC node, against about nothing on master. A copied cached node keeps the values it was copied with, as on master. - The lazy state is copied as it is once complete, so a copy's _children and its state stay one mapping and add_child on the copy is listed. - add_child on a fully read node no longer changes what attribute reads return on its other copies (the resolved children and the items were one dictionary). Items are read from the complete mapping, so direct writes to _children are items, as before. - A parameter node's at-instant cache is an _AtInstantCache, a dict that copies and pickles from a one-step copy of its items. Copies stay safe while other threads read, and parameter nodes sharing one cache still share its copy. ParameterNode's own __getstate__ and __copy__ are gone. - ParameterScale sets each loaded bracket's parent, as clone already did and as #660 does. Parameter.update and add_child below a bracket now clear the caches above the scale, so a reform of a bracket reads the same whatever was calculated before it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes from the second review round:
- homogenize_parameter_structures links the nodes it puts in place of
missing breakdown values to their parent node. They were set straight
into children with no parent, so an update below one cleared caches
only up to it, and whether a later read saw the update depended on
what had been read before (policyengine-us has 9,811 such nodes).
- A subclass of ParameterNodeAtInstant that overrides add_child gets
the up-front build as before: every child added into _children, which
is also the mapping items are read from. An add_child that wrote
_children directly lost its items.
- A node at a date that does not exist ("2015-02-29", "0000-01-01") is
built up front, so it fails as before. periods.instant reads these;
str(Instant), which a scale's build calls, refuses them.
- A node's children are the ones node.children.items() gives, as the
up-front build iterated them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes from the third review round: - Any subclass of ParameterNodeAtInstant gets the up-front build, not only one that overrides add_child. A subclass whose __getattr__ or __getitem__ answers names that are not children returned its fallback for real children not read yet. - ParameterNode._get_at_instant replaces a plain dict cache, with its entries, by an _AtInstantCache before adding to it. policyengine-us's copy_parameter_nodes sets plain dict caches, as do trees pickled by an earlier core, and copying or pickling such a tree while another thread resolved a child raised "dictionary changed size during iteration". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fixes from the fourth review round: - ParameterNode._get_at_instant stores the node it built into the cache the node holds after building it, as before. If code run while the node was built replaced the cache (policyengine-us's isolate_parameter_tracing does, to drop an earlier request's nodes), the old entries came back. - A complete node's resolved children are no longer the same dictionary as its items, so a write into _children changes items only, as when every child was built up front, and attributes keep what the parameters give. - A pickled _AtInstantCache is a plain dict, so trees pickled by this version load in earlier versions of core when they hold no nodes at instant. - The class docstring says that a child that fails to build fails when read, and that a pickle of a node at instant holds this version's lazy state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Problem
ParameterNodeAtInstant.__init__built every descendant up front. The first time a tax-benefit system was asked for its parameters at a date, it built that date's whole tree of nodes at instant, andParameterNode._at_instant_cache(plusTaxBenefitSystem._parameters_at_instant_cache) kept it for the system's lifetime. Every reform system, whose caches start empty, paid again.Change
Max's commits:
__getattr__,__getitem__) and keeps it. It resolves all of its direct children only when something needs all of them:_children, iteration,repr,dir, vectorial indexing. A child read as an attribute becomes a plain instance attribute, so later reads cost what they did.Added on top, each with failing tests first:
ParameterNode.add_childclears the cached nodes at instant of the node and of every node above it, which were built without the new child. It walks up without recursing and stops at a node it has already visited. While a node loads, a child namedparent(policyengine-us hasgov.hhs.medicaid.eligibility.categories.parent) replaces the node'sparentattribute and points back at it. The walk clears before the child is set as an attribute and only clears a realdict, so a child named_at_instant_cacheis fine too.ParameterScalesets each loaded bracket'sparent, asclonealready did (the same line as Link scale brackets and homogenized nodes to their parents #660).homogenize_parameter_structureslinks the nodes it puts in place of missing breakdown values (policyengine-us has 9,811). Without these links,Parameter.updatebelow them cleared caches only up to them. On master that left stale values; with lazy nodes, whether a later read saw the update depended on what had been read before. The links also letmark_as_modifiedreach the scale and the nodes above it, as in a cloned tree, whichSimulation.check_macro_cachereads.__dict__, its lazy state and the_at_instant_cacheof the parameter nodes below it. Copying or pickling those dictionaries directly raisedRuntimeError: dictionary changed size during iterationif a child was resolved meanwhile.__dict__._AtInstantCache: adictwhose copy or pickle starts from a one-step copy of its items and still keeps it one object for nodes that share it._get_at_instantreplaces a plaindictcache set from outside core (policyengine-us'scopy_parameter_nodessets one) before adding to it.Testjob of this PR's earlier CI run.parameters(None)reads "None"; "2015-02-29"), is built up front, so it fails, or succeeds, as before.Simulation._calculatereadsparameters(period).gov.abolitions[...]withperiodNone and ignores errors; a lazy node there would have read the latest values.ParameterNodeAtInstantgets the up-front build, so itsadd_child,__getattr__and__getitem__see every child as before.node.children.items()gives, as the up-front build iterated them.os.register_at_fork). A process forked while another thread held the lock could never resolve a child.Same values as before
ParameterNodetree directly.Parameter.update,add_childor a cache reset read the change, as before. With the parent links above, that now includes changes below scale brackets and homogenized nodes, where master kept serving stale values.TaxBenefitSystem._parameters_at_instant_cache, which nothing clears afterParameter.updateorSimulation.apply_reform(Define how Core's parameter caches are owned, invalidated, copied and shared #621, problem 1). Formulas read through the root's_at_instant_cache, which those clear. Removing this would take either the up-front build or Define how Core's parameter caches are owned, invalidated, copied and shared #621's rule that trees don't change after construction.dict.policyengine-us's
spm.pypolicyengine_us/spm.py(maine283430b2e) handles these caches directly:_at_instant_cache;_parameters_at_instant_cachebetween policies;copy_parameter_nodes;All of this works unchanged. A lazy node resolves children through the shared child nodes' caches, so it holds the same objects the up-front build did.
test_parameter_node_at_instant_systems.pyfollows each pattern in Core. A script against policyengine-us main read identical values and parameter-tree hashes on master and this branch (20 of 20):isolate_parameter_tracingthrough a traced simulation, a branch sharing a policy and then a reform that detaches it,pin_spm_policy,clone_spm_system,copy_parameter_nodeswith an update, andshare_spm_policy.Interaction with #660
#660 also changes
ParameterNode.add_child: it moves the body into_link_childand addsreplace_child. Whichever PR merges second has one textual conflict there. To resolve it, keep #660's_check_child_type(child)andself._link_child(name, child), and call_clear_at_instant_caches_upward(self)just before_link_child. Thebracket.parent = selfline is the same in both and merges cleanly, and so does the homogenization change (this PR links parents after homogenizing; #660 usesreplace_childwhile homogenizing). This branch's tests all pass on the merge of the two.With #660, children named like node attributes aren't set as attributes, so the loop the
add_childwalk guards against no longer happens, and its parent links make this PR's homogenization pass find nothing to link.Relation to #621
#621 proposes rules for Core's parameter caches and suggested lazy nodes wait for them. This PR doesn't wait. It keeps every value a node requested after a change reads, and it handles the three problems #621 lists for this PR's earlier version:
add_childclears now, and brackets and homogenized nodes have parents. The held-node case above is what remains.RuntimeError. Fixed, with deterministic tests.The rest of #621 (freezing trees, removing the system-level cache, not copying caches) is unchanged and still applies.
Measurements
Re-measured on 2026-11-17 against master 175e7ee. Setup:
e283430b2e(2.144.4) on Python 3.13, with its default dataset (populace_us_2024.h5);subsample(n=2862, seed=0), which gives 2,490 households;household_net_incomefor 2026;Microsimulation(), the subsample, then the calculation.Microsimulation(reform={"gov.irs.deductions.standard.amount.SINGLE": {"2026-01-01.2100-12-31": 20000}})(policyengine-us builds a new system), the same subsample and the same calculation.Outputs are identical. Every stored array of both simulations and their branches matches master: 428,303 arrays, hashed over dtype, shape and bytes. So do the weighted totals.
Times. Runs were interleaved (master, branch, master, ...) on a shared machine with a load average of about 20 on 10 cores, so single runs vary by several seconds.
Memory. Peak resident memory couldn't be compared on this machine: the medians were 6.94 GB against 5.54 GB in one set of runs, and 5.43 GB against 5.44 GB in the next.
Node objects were counted by wrapping
ParameterNodeAtInstant.__init__in a separate run, and live objects withgcafter each phase.Max's measurements (2026-10-02)
Measured on policyengine-us (132,396 parameter descendants) on master b78b0ba:
A YAML run touches every year it tests and every month of a monthly variable, so the shared system and each reform system all pay this many times over. It was the third cause of the 118 GB
policyengine-core testruns on 2026-10-02, next to the test-runner causes fixed in #569; the two PRs are independent. Combined with #569 on the two policyengine-us CI batches that failed on 16 GB runners, peak memory fell from 16.3 GB to 7.0 GB (NY/OH/OK batch) and from 17.1 GB to 6.4 GB (USDA batch): #569 (comment).For downstream packages
mainrely on the up-front build, and will fail once it takes a core with this change. Simulation outputs are not affected.tests/code_health/test_labour_supply_parameters_are_read.pyrecords parameter reads by patchingParameterNodeAtInstant.__getattribute__. A first read of a child now goes through__getattr__, andobject.__getattribute__(node, "_children")raises until the node is read in full. So it misses reads, and the test reports four labour supply parameters as unread.test_uprating_inputs_reach_data.py::test_a_failing_probe_leaves_the_system_and_its_caches_aloneexpects reading the root at an instant to fill descendants' caches.prewarm_parameter_cachescallsget_parameters_at_instantfor many instants so the memory snapshot holds built trees. It now builds only each root; the first requests resolve what they read.Tests
Added or changed, run file by file:
tests/core/parameters/test_parameter_node_at_instant_lazy.pytest_parameter_node_at_instant_property.pytest_parameter_node_add_child_caches.pytest_parameter_node_at_instant_copies.pytest_parameter_node_at_instant_systems.pyAll 235 pass, and they also pass on a merge with #660. The rest of the suite wasn't run locally. Five rounds of independent adversarial review: the first four found bugs, each fixed with a failing test first; the fifth found none in Core.
Credit: the lazy resolution, its thread safety and its property tests are Max Ghenis's work.
🤖 Generated with Claude Code