Skip to content

Link scale brackets and homogenized nodes to their parents - #660

Open
anth-volk wants to merge 5 commits into
masterfrom
link-parameter-parents
Open

anth-volk wants to merge 5 commits into
masterfrom
link-parameter-parents

Conversation

@anth-volk

@anth-volk anth-volk commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Parameter.update walks upward through parent to clear cached at-instant nodes and to mark ancestors as modified. Two places built parameter trees without those links, so a tree loaded from files behaved differently from its clone(), which links every node:

  • ParameterScale.__init__ never set bracket.parent. It now sets it to the scale.

  • homogenize_parameter_node stored the ParameterNode it builds for a further breakdown level in children without linking it, and the holder's attribute kept pointing at the placeholder Parameter. It now uses a new private method, ParameterNode._replace_child, which:

    • links the replacement and sets the attribute;
    • clears cached values of the node and its ancestors;
    • leaves the previous child as it is, and marks nothing modified;
    • rejects the node itself, an ancestor or a non-string name, checking every argument before changing the tree.

    The hasattr(node, "children") check is now isinstance(node, ParameterNode).

  • The country template reform that appends a bracket now links it to its scale.

add_child, _replace_child and clone no longer set a child as an attribute when its name matches one of ParameterNode's own instance attributes (parent, children, name, modified, …) or a method or class attribute of the node's classes (clone, merge, …). Metaclass attributes such as register or mro are not blocked. A blocked child stays reachable through children[...] and at-instant lookups, so parameters(period).x.parent is unchanged.

Before this change:

  • a child named children broke loading when another child followed it, and broke clone() otherwise;
  • a root node's child named parent replaced the root's parent link;
  • a placeholder named parent added by homogenization replaced the holder's parent link;
  • clone() of a node with a child named modified set modified to that child, which is truthy.

Effect on the policyengine-us 2.144.2 tree

core 3.36.1 this branch
Nodes with parent = None (loaded tree) 18,246 (8,435 brackets, 9,811 homogenized nodes) 0
Node attributes not pointing at the child in children 9,820 9 (8 children named parent, 1 named modified; both names are blocked on purpose)
Nodes below the root marked modified after updating gov.irs.credits.ctc.amount.base[0].amount, loaded tree / clone 2 / 9 8 / 8
  • The root's flag: the root itself is already marked modified after the US build on both versions, before any update, so the table leaves it out.
  • The extra node on the 3.36.1 clone: its ninth node is gov.states.mt.dphhs.liheap.payment, reported modified because its child named modified had replaced the flag.
  • Effect on the simulation cache: Simulation.check_macro_cache (the only reader of modified) now sees bracket updates under whole-node dependencies that contain scales, such as gov.usda.snap and gov.ssa, which snap and snap_if_takes_up declare.
  • Stale values: a bracket update now clears stale cached ancestor values. With 3.36.1, after an in-place edit of gov.irs.credits.ctc.amount.base[0].amount following a first simulation, the next simulation still gave the old ctc_maximum (4,400). With this branch it gives the new one (6,400).

On the policyengine-us tree, the only names this change no longer sets as attributes are parent (Medicaid eligibility categories) and modified (Montana LIHEAP payment), all from YAML files. For directory-loaded trees, 3.36.1 already gave those attributes the holding node and the False flag, because the holder's add_child and __init__ re-assigned them afterwards. So attribute reads of those names on a loaded tree are unchanged. A scan of the parameter folder of a policyengine-uk checkout found no blocked names.

Build time

  • Wall clock: builds of CountryTaxBenefitSystem() did not get measurably slower. These alternating runs were made before the last commit:
    • 5 runs: 3.36.1 23.5–31.6 s (median 24.3 s), this branch 23.6–31.4 s (median 25.7 s);
    • a separate set of 4 runs: means of 12.0 s and 12.1 s.
  • Last commits: they only make _replace_child cheaper and private. I could not repeat the wall-clock runs on a quiet machine after it.
  • Profiler, current code: _replace_child (19,622 calls per build, including the cache clearing) takes 0.43 s cumulative, and the blocked-name check 0.49 s, in a profiled build that took 228 s on a heavily loaded machine.

Changes callers will see

  • Cache reuse: updating a bracket now marks the scale and all its ancestors modified. Simulation.check_macro_cache therefore stops reusing cached results for variables that declare those nodes as dependencies. This is the intended fix.
  • Homogenized positions: a position that homogenization replaced now resolves, by attribute, to the replacement ParameterNode instead of the placeholder Parameter. For example, amount.B("2021-01-01") returns a node at that instant rather than 0, and amount.B.update(...) raises. This affects 9,811 US attributes, and at-instant lookups are unchanged.
  • Cloned brackets: a bracket's clone() now keeps the original scale as parent, as cloning any other non-root node already did. Updating a standalone cloned bracket therefore clears the original tree's caches and marks it modified.
  • policyengine-us test: policyengine_us/tests/test_pinned_tbs_sharing.py::test_an_edit_to_a_loaded_tree_reaches_the_next_simulation asserts loaded.parameters.gov.irs.credits.eitc.max.brackets[1].parent is None, the unlinked state of released core. It will fail on that precondition with this change. That line, and the comments in that file that describe unlinked brackets, need updating when policyengine-us moves to a core release containing this change.
  • .py parameter labels: .py's utils/parameter_labels.py walks parent to find the ancestor with breakdown metadata, so it now generates labels for homogenized US parameters. Running that module from a .py checkout against both versions: of all 128,462 US parameters, those without a label fall from 36,806 to 2,606. No existing label changes, and there are no errors.
  • Appending brackets outside core: code that appends a bracket to scale.brackets must set bracket.parent = scale itself, as the template reform now does.
  • Deep copies: copy.deepcopy of a single bracket now copies the whole tree through parent, as it already did for every other node type. I found no code in core, policyengine-us, policyengine-uk or .py that deep-copies a single bracket.

Limits

  • Blocked names are computed once per class. A method added to a node class after its first use is not blocked.
  • Subclass instance attributes: only ParameterNode's own instance attributes are protected, not instance attributes that a subclass sets itself. No subclass in core or policyengine-us sets one.
  • Template reform caches: Reform.modify_parameters deep-copies at-instant caches, and the template reform edits values in place without clearing them. That is unchanged here.

Interaction with #570

#570 (resolve parameter nodes at instant on first read) changes parameter_node_at_instant.py, its tests, a test fixture and a changelog fragment. This PR shares no files with it.

  • Lazily resolved at-instant nodes read the source node's children on first access. They depend on ParameterNode._at_instant_cache being cleared upward when a parameter changes, and this change makes that clearing reach every ancestor of brackets and homogenized nodes.
  • ParameterNodeAtInstant.add_child (on master, and changed by Resolve parameter nodes at instant on first read #570) still sets each child as an attribute without the blocked-name check. This PR leaves at-instant nodes unchanged.

Tests

  • New tests: tests/core/parameters/test_parameter_tree_parent_links.py and tests/core/parameters/operations/test_homogenization_parent_links.py, 28 tests in all.
    • All pass on this branch.
    • 25 fail on released core 3.36.1, whose versions of the four source files this branch changes are identical to master's.
    • The 3 that pass there guard behaviour that already worked in released core: children named after metaclass attributes stay attributes, and placeholders added for missing single-level breakdown values are linked.
  • Commands run: only these two test files, plus ruff format --check and ruff check. I did not run the full suite.
  • Real-tree check: run on the policyengine-us 2.144.2 tree from PyPI, in separate environments with released core 3.36.1 and with this branch (numbers above). Each was run with python -P from outside the checkout, so the "before" environment imports the released core.
  • Self-review: the author's workflow ran six review rounds, each with a separate reviewer. The bugs they confirmed are fixed here, each with a test written first:
    • _replace_child left cached values stale.
    • The template reform's bracket was unlinked.
    • Children named after metaclass attributes were blocked.
    • Three cases where _replace_child detached a previous child it should not have, or failed partway. They are fixed by no longer detaching.
    • _replace_child accepted the node itself, an ancestor or a non-string name and left a changed tree behind.
  • Private method: _replace_child was public during review. It is now private because homogenization is its only caller, so this PR adds no public API.

🤖 Generated with Claude Code

anth-volk and others added 5 commits November 17, 2026 13:22
ParameterScale.__init__ now sets each bracket's parent to the scale, and
homogenization links any ParameterNode it builds to replace a placeholder
Parameter (through the new ParameterNode.replace_child), so a loaded tree
behaves like its clone when Parameter.update clears cached at-instant values
and marks ancestors as modified. The country template reform links the
bracket it appends.

A child whose name matches one of the node's own attributes or methods
(for example parent, children or modified) is no longer set as an
attribute, so it can no longer replace the node's own state.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
replace_child now detaches the previous child only while it still belongs
to the node, so a child moved under the replacement keeps its link.

The check for names that must not be set as child attributes now looks
only in the node's classes, not their metaclasses, so children named
register or mro stay reachable as attributes, and the blocked names are
computed once per class.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A Parameter or ParameterScale stored in children directly has no parent
attribute, so replace_child raised after linking the replacement and before
clearing cached values. It now reads the attribute with a default.

Tests for reserved child names now check a root node and its clone, where an
overwrite stays visible, and the add_child docstring says which names are not
set as attributes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
replace_child now rejects a non-string name, the node itself or one of its
ancestors before changing anything, so a rejected call leaves the tree as
it was instead of a cycle or a partial update.

It no longer detaches the previous child. Detaching needed a scan of the
holder's children to tell whether the child was still held under another
name, which made homogenization quadratic for holders with many replaced
children. The previous child now keeps its link, as homogenization's
replaced placeholders did before.

Corrects the comment in the reserved-name test: __init__ resets modified on
a root, so only the clone shows an overwrite of that attribute.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Homogenization is its only caller, so the fix adds no public API. Removes
the changelog fragment that announced it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk added a commit that referenced this pull request Nov 17, 2026
Fixes from the first review round:

- A deep copy or pickle of a node at instant resolves every child first,
  so it is a snapshot that holds no parameter nodes, as when every child
  was built up front. Before, a partly read node carried its unread
  parameter nodes, and through their parents the whole tree: 128 MB and
  22 s to deep-copy a policyengine-us CTC node, against about nothing on
  master. A copied cached node keeps the values it was copied with, as on
  master.
- The lazy state is copied as it is once complete, so a copy's
  _children and its state stay one mapping and add_child on the copy is
  listed.
- add_child on a fully read node no longer changes what attribute reads
  return on its other copies (the resolved children and the items were
  one dictionary). Items are read from the complete mapping, so direct
  writes to _children are items, as before.
- A parameter node's at-instant cache is an _AtInstantCache, a dict that
  copies and pickles from a one-step copy of its items. Copies stay safe
  while other threads read, and parameter nodes sharing one cache still
  share its copy. ParameterNode's own __getstate__ and __copy__ are gone.
- ParameterScale sets each loaded bracket's parent, as clone already did
  and as #660 does. Parameter.update and add_child below a bracket now
  clear the caches above the scale, so a reform of a bracket reads the
  same whatever was calculated before it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant