Skip to content

Return plain arrays to Microsimulation calls made inside a calculation - #661

Open
anth-volk wants to merge 4 commits into
masterfrom
unweighted-internal-calls
Open

anth-volk wants to merge 4 commits into
masterfrom
unweighted-internal-calls

Conversation

@anth-volk

@anth-volk anth-volk commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator
  • make format && make documentation has been run. (ruff format and ruff check were run on the changed files only; the documentation was not built, since no documentation page changed.)

Bug fix

  • Regression test added
  • Regression test passing

Fixes #658.

What changes

Microsimulation.calculate, calculate_add and calculate_divide return a weighted MicroSeries, and calculate decodes enums to member names. Core also calls these methods for its own reads while it calculates: each term of adds and subtracts, a defined_for mask, each month of a monthly variable read over a year, the year of a yearly variable read over a month, and the terms of ADD and DIVIDE. Each such call computed household_weight, projected it to the variable's entity and built a MicroSeries, and core then discarded the weights before storing the value (_cast_formula_result). Decoding broke enum reads with ADD and DIVIDE (#658).

This PR changes only policyengine_core/simulations/microsimulation.py:

  • Detection. A private ContextVar holds a record of the innermost Microsimulation calculate, calculate_add or calculate_divide call running in the current context; each record links to the one for the call that encloses it. A call that arrives while one of these calls is running is a call from inside a calculation. When a call ends, its record is marked as finished. A context copied during a call (for a worker thread, say) is therefore inside while that call, or one enclosing it, is still running, or while a call of its own runs there, and outside afterwards. A new thread starts with an empty context, so its calls count as outside (unless Python runs with -X thread_inherit_context=1, the default on free-threaded builds, which gives the thread a copy). Core's _calculation_frames and the tracer stack are not used.
  • Defaults. All three methods now default to use_weights=None and decode_enums=None.
    • Inside a calculation, None gives a plain NumPy array with enums left as codes, as a Simulation returns.
    • Outside, None gives the previous defaults: calculate is weighted and decoded; calculate_add and calculate_divide are weighted and not decoded (Fix formula reads with ADD or DIVIDE in a Microsimulation #647).
    • An explicit argument wins in both cases.
  • ADD and DIVIDE. Their term reads arrive while the call is running, so they get codes, and the result is decoded once when decode_enums=True. A twelfth of a code from a month DIVIDE is returned undecoded, as a Simulation returns it.
  • get_weights reads household_weight with decode_enums=False.
  • simulation.py's call sites are unchanged. Calling the base class from there would skip subclass overrides of calculate.

Effect

One weighted series is built per top-level call. Before this change, for example, a top-level calculate of a yearly variable with a monthly adds term built 13; a 500-household policyengine-us household_net_income built 35,368. On 100 policyengine-us households (warm household_net_income, 3 fresh processes), a variant of this change that detected internal calls through _calculation_frames took 3.57 s, against 6.84 s on master. It classifies every consumer call the same way as this change; this change's own timing was not measured on that setup. On 500 households, an earlier prototype of this change's detection took the baseline from 10.2 s to 6.3 s, measured on a loaded machine.

Checked against every consumer

Each consumer was run on master 175e7ee and with this change (or a variant that classifies every consumer call the same way), comparing every stored array's values, dtype, writability and keys, and every top-level value and weight. Consumers checked: policyengine-us 2.144.1, policyengine-uk 2.179.1, policyengine-canada 0.99.2, policyengine-il 0.2.2, policyengine-ng 0.6.2, policyengine-italy 0.16.3, and .py (policyengine package) at main 47d1d8bf (code read, not run).

  • No value that computes on master changes. A copy of this change's detection gave 2,898,199 stored arrays bit-identical to the _calculation_frames variant, which in turn differed from master only as listed below. Runs covered a targeted 1,140-household policyengine-us file for 2024 and 2027, current law and a CTC reform; 121 policyengine-us YAML cases (402 outputs) replayed inside a Microsimulation, 42 more at 2027, and the WI WHEAP file; and one synthetic household each in Canada, Israel, Nigeria and Italy. Separately, 500 policyengine-us households gave 508,321 stored arrays bit-identical to master.
  • policyengine-uk never reaches core's Microsimulation: it does not subclass it, and a counter recorded 0 calls.
  • Direct calculate calls in consumer formulas. 22 calls in 20 policyengine-us formula files call calculate on a simulation, branch or baseline. Examples: tax_liability_if_itemizing / _not_itemizing, tax_unit_itemizes, the NY and NYC EITC, the VA, DE, HI, ID and MO pairs, claim of right, co_adult_financial_medicaid_enrolled, medicaid_enrolled_for_ssi_state_supplement, spm_unit_allocated_housing_subsidy and marginal_tax_rate_helpers. They now get an ndarray rather than a MicroSeries. Each reads a float or bool variable and uses the result element by element, returns it, or passes it to map_result or set_input. None uses .values, .weights, .sum() or pandas indexing on the result, and every one ran in a Microsimulation in the comparisons above.
  • Calls that raised on master now compute.
    • wi_wheap_categorically_eligible (policyengine-us) raised microdf's "Cannot propagate weights unambiguously" in a Microsimulation on master (8 of 8 YAML cases); all 8 now pass.
    • Canada, Israel, Nigeria and Italy define no household_weight, so most internal reads raised there. Now 294 of 405 calls compute in Canada, 27 of 40 in Israel, 27 of 36 in Nigeria and 38 of 60 in Italy. Weighted calls from outside a calculation still raise there.
  • Stored arrays that were read-only pandas views are now writable NumPy arrays with the same bytes: about 1,000 to 3,300 per policyengine-us run, from period conversion and adds.
  • Some branch household_weight arrays are no longer calculated or copied, because a formula's branch.calculate is no longer weighted. Where they still exist, the values are identical.
  • Enums. No consumer has an adds/subtracts list, a defined_for or a period-converted read that involves an Enum variable, and no consumer formula reads with options=[ADD] or [DIVIDE], so no stored enum code changes. The Enum reads with ADD or DIVIDE raise or return names in a Microsimulation #658 reads now match Simulation. One top-level result changes: calculate_divide(enum_variable, year) left at decode_enums=None now returns codes, as Fix formula reads with ADD or DIVIDE in a Microsimulation #647's False default intends; master returned names. No consumer makes that call.

Not covered by those checks: full national datasets; .py at runtime; this change's exact detection on policyengine-uk (which cannot reach it) and its timing; Simulation.calculate_output has no caller in any consumer.

Other behaviour to note

  • Formula code that calls these methods directly now gets a NumPy array with enum codes, not a weighted MicroSeries with member names. A formula that took .sum() of a direct simulation.calculate(...) read, compared a direct enum read with a member name, or used .values or .weights on it would now get a different result or an error. The consumer check above found no such code; policyengine-us's direct calls return the result or use it element by element. The same applies to calculate_dataframe, to_input_dataframe and derivative when they are called inside a formula: their inner reads count as inside, so enum columns hold codes.
  • Detection follows contextvars contexts, not threads. A call run through a context captured before the calculation started (for example Context.run on a context copied at import time, or an asyncio.Runner first used outside any calculation) counts as outside, even in the same thread, and gets weighted values, as on master.
  • Explicit use_weights=False from outside a calculation is now honoured everywhere. On master, without map_to, calculate_add, calculate_divide and a calculate of a FLOW variable over a period of another length (a monthly variable over a year, a yearly one over a month) returned a weighted MicroSeries even with use_weights=False, because their inner reads were weighted. They now return a plain array, so .sum() is the unweighted sum (for example 624 rather than 10,140 in the new tests' fixture). A review of policyengine-us, policyengine-uk, .py and the Public API found no caller that depends on the old result.
  • Reads inside a calculation return what a Simulation returns, including the stored array itself. On master, some of these reads in a Microsimulation returned a copy as a side effect of building a MicroSeries: a period-converted read, a read with options=[ADD] or [DIVIDE] over a year, and a formula's direct calculate. Values stored from period conversion (ADD from months to a year, DIVIDE from a year to a month) were stored as read-only arrays built from a pandas Series on master; they are now stored writable, with the same bytes. A formula that writes in place into such a read now changes the stored value, as it already does in a Simulation and for plain same-period reads. A scan of policyengine-us found no formula that writes into these reads.
  • Passing use_weights=None or decode_enums=None explicitly used to mean unweighted and not decoded; it now means the default for the caller (weighted and, for calculate, decoded from outside a calculation). No caller was found that passes None.
  • calculate_dataframe is unchanged: it still returns a weighted MicroDataFrame with enum names, and still builds weights once per column, which it discards, plus once for the frame (4 for 3 columns in the new tests' fixture, against 33 on master). Removing the per-column weights without changing its enum decoding would need a separate way to turn weights off, which this PR does not add.
  • A Microsimulation used inside a formula of a plain Simulation (not a Microsimulation) still gets weighted results, as on master: only a running Microsimulation call marks a call as internal. No consumer does this.

Tests

New tests/core/test_microsimulation_internal_calls.py, with a fixture system in tests/fixtures/microsimulation_internal_calls.py:

  • one test for each way a call can arrive: a formula reading its own simulation (population read, direct calculate, fast-cache re-read, calculate_add, year and month calculate_divide), trace off and on; a branch, trace off and on; a reform's baseline; a Microsimulation built in a Microsimulation's formula and in a plain Simulation's formula; a worker thread with and without a copied context; a context copied in a formula and used after the calculation ends; a call in a copied context that keeps running after the call it was copied from has ended; a context copied during a nested call and used while an enclosing call still runs; the fast-cache re-read; explicit arguments inside and at the top level;
  • top-level calculate, calculate_add, calculate_divide, calculate_output and calculate_dataframe are still weighted, and each top-level call builds exactly one weighted series (for adds/subtracts, defined_for, a monthly variable over a year, a yearly variable over a month, ADD and DIVIDE), with household_weight calculated once;
  • after a formula raises (RuntimeError, a BaseException, trace on), an ADD or DIVIDE term raises, a formula catches a branch's error, or a branch input set during a calculation makes core run it a second time, calls from outside a calculation are weighted again;
  • a Microsimulation stores and returns what a Simulation does: same dtype, bytes and writability for every stored array;
  • the Enum reads with ADD or DIVIDE raise or return names in a Microsimulation #658 enum reads (ADD over a year, DIVIDE over a year and a month, from the top level and from formulas) match a Simulation, and top-level enum calculate_add / calculate_divide decode when asked.

test_enum_read_with_divide_over_a_year_gives_member_names_in_a_microsimulation recorded the #658 bug; it now checks that the codes match a Simulation. test_microsimulation_calculate_signatures.py checks the new None defaults.

Run: each new and modified test, selected individually by node ID (all pass; the thread and copied-context tests also with -X thread_inherit_context=1), and ruff format / ruff check on the changed files. The rest of the suite was not run locally; CI runs it.

🤖 Generated with Claude Code

anth-volk and others added 4 commits November 17, 2026 13:35
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ile its own call runs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… runs

Fix tests that did not reach the fast cache or the rerun after an input
change, wait for the worker thread's call to start, and narrow the
changelog fragments to what changed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rectly

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enum reads with ADD or DIVIDE raise or return names in a Microsimulation

1 participant