Repository navigation
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ile its own call runs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… runs Fix tests that did not reach the fast cache or the rerun after an input change, wait for the worker thread's call to start, and narrow the changelog fragments to what changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rectly Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
make format && make documentationhas been run. (ruff formatandruff checkwere run on the changed files only; the documentation was not built, since no documentation page changed.)Bug fix
Fixes #658.
What changes
Microsimulation.calculate,calculate_addandcalculate_dividereturn a weightedMicroSeries, andcalculatedecodes enums to member names. Core also calls these methods for its own reads while it calculates: each term ofaddsandsubtracts, adefined_formask, each month of a monthly variable read over a year, the year of a yearly variable read over a month, and the terms of ADD and DIVIDE. Each such call computedhousehold_weight, projected it to the variable's entity and built aMicroSeries, and core then discarded the weights before storing the value (_cast_formula_result). Decoding broke enum reads with ADD and DIVIDE (#658).This PR changes only
policyengine_core/simulations/microsimulation.py:ContextVarholds a record of the innermostMicrosimulationcalculate,calculate_addorcalculate_dividecall running in the current context; each record links to the one for the call that encloses it. A call that arrives while one of these calls is running is a call from inside a calculation. When a call ends, its record is marked as finished. A context copied during a call (for a worker thread, say) is therefore inside while that call, or one enclosing it, is still running, or while a call of its own runs there, and outside afterwards. A new thread starts with an empty context, so its calls count as outside (unless Python runs with-X thread_inherit_context=1, the default on free-threaded builds, which gives the thread a copy). Core's_calculation_framesand the tracer stack are not used.use_weights=Noneanddecode_enums=None.Nonegives a plain NumPy array with enums left as codes, as aSimulationreturns.Nonegives the previous defaults:calculateis weighted and decoded;calculate_addandcalculate_divideare weighted and not decoded (Fix formula reads with ADD or DIVIDE in a Microsimulation #647).decode_enums=True. A twelfth of a code from a month DIVIDE is returned undecoded, as aSimulationreturns it.get_weightsreadshousehold_weightwithdecode_enums=False.simulation.py's call sites are unchanged. Calling the base class from there would skip subclass overrides ofcalculate.Effect
One weighted series is built per top-level call. Before this change, for example, a top-level
calculateof a yearly variable with a monthlyaddsterm built 13; a 500-household policyengine-ushousehold_net_incomebuilt 35,368. On 100 policyengine-us households (warmhousehold_net_income, 3 fresh processes), a variant of this change that detected internal calls through_calculation_framestook 3.57 s, against 6.84 s on master. It classifies every consumer call the same way as this change; this change's own timing was not measured on that setup. On 500 households, an earlier prototype of this change's detection took the baseline from 10.2 s to 6.3 s, measured on a loaded machine.Checked against every consumer
Each consumer was run on master 175e7ee and with this change (or a variant that classifies every consumer call the same way), comparing every stored array's values, dtype, writability and keys, and every top-level value and weight. Consumers checked: policyengine-us 2.144.1, policyengine-uk 2.179.1, policyengine-canada 0.99.2, policyengine-il 0.2.2, policyengine-ng 0.6.2, policyengine-italy 0.16.3, and .py (
policyenginepackage) atmain47d1d8bf (code read, not run)._calculation_framesvariant, which in turn differed from master only as listed below. Runs covered a targeted 1,140-household policyengine-us file for 2024 and 2027, current law and a CTC reform; 121 policyengine-us YAML cases (402 outputs) replayed inside aMicrosimulation, 42 more at 2027, and the WI WHEAP file; and one synthetic household each in Canada, Israel, Nigeria and Italy. Separately, 500 policyengine-us households gave 508,321 stored arrays bit-identical to master.Microsimulation: it does not subclass it, and a counter recorded 0 calls.calculatecalls in consumer formulas. 22 calls in 20 policyengine-us formula files callcalculateon a simulation, branch or baseline. Examples:tax_liability_if_itemizing/_not_itemizing,tax_unit_itemizes, the NY and NYC EITC, the VA, DE, HI, ID and MO pairs, claim of right,co_adult_financial_medicaid_enrolled,medicaid_enrolled_for_ssi_state_supplement,spm_unit_allocated_housing_subsidyandmarginal_tax_rate_helpers. They now get anndarrayrather than aMicroSeries. Each reads a float or bool variable and uses the result element by element, returns it, or passes it tomap_resultorset_input. None uses.values,.weights,.sum()or pandas indexing on the result, and every one ran in aMicrosimulationin the comparisons above.wi_wheap_categorically_eligible(policyengine-us) raised microdf's "Cannot propagate weights unambiguously" in aMicrosimulationon master (8 of 8 YAML cases); all 8 now pass.household_weight, so most internal reads raised there. Now 294 of 405 calls compute in Canada, 27 of 40 in Israel, 27 of 36 in Nigeria and 38 of 60 in Italy. Weighted calls from outside a calculation still raise there.adds.household_weightarrays are no longer calculated or copied, because a formula'sbranch.calculateis no longer weighted. Where they still exist, the values are identical.adds/subtractslist, adefined_foror a period-converted read that involves an Enum variable, and no consumer formula reads withoptions=[ADD]or[DIVIDE], so no stored enum code changes. The Enum reads with ADD or DIVIDE raise or return names in a Microsimulation #658 reads now matchSimulation. One top-level result changes:calculate_divide(enum_variable, year)left atdecode_enums=Nonenow returns codes, as Fix formula reads with ADD or DIVIDE in a Microsimulation #647'sFalsedefault intends; master returned names. No consumer makes that call.Not covered by those checks: full national datasets; .py at runtime; this change's exact detection on policyengine-uk (which cannot reach it) and its timing;
Simulation.calculate_outputhas no caller in any consumer.Other behaviour to note
MicroSerieswith member names. A formula that took.sum()of a directsimulation.calculate(...)read, compared a direct enum read with a member name, or used.valuesor.weightson it would now get a different result or an error. The consumer check above found no such code; policyengine-us's direct calls return the result or use it element by element. The same applies tocalculate_dataframe,to_input_dataframeandderivativewhen they are called inside a formula: their inner reads count as inside, so enum columns hold codes.contextvarscontexts, not threads. A call run through a context captured before the calculation started (for exampleContext.runon a context copied at import time, or anasyncio.Runnerfirst used outside any calculation) counts as outside, even in the same thread, and gets weighted values, as on master.use_weights=Falsefrom outside a calculation is now honoured everywhere. On master, withoutmap_to,calculate_add,calculate_divideand acalculateof a FLOW variable over a period of another length (a monthly variable over a year, a yearly one over a month) returned a weightedMicroSerieseven withuse_weights=False, because their inner reads were weighted. They now return a plain array, so.sum()is the unweighted sum (for example 624 rather than 10,140 in the new tests' fixture). A review of policyengine-us, policyengine-uk, .py and the Public API found no caller that depends on the old result.Simulationreturns, including the stored array itself. On master, some of these reads in aMicrosimulationreturned a copy as a side effect of building aMicroSeries: a period-converted read, a read withoptions=[ADD]or[DIVIDE]over a year, and a formula's directcalculate. Values stored from period conversion (ADD from months to a year, DIVIDE from a year to a month) were stored as read-only arrays built from a pandas Series on master; they are now stored writable, with the same bytes. A formula that writes in place into such a read now changes the stored value, as it already does in aSimulationand for plain same-period reads. A scan of policyengine-us found no formula that writes into these reads.use_weights=Noneordecode_enums=Noneexplicitly used to mean unweighted and not decoded; it now means the default for the caller (weighted and, forcalculate, decoded from outside a calculation). No caller was found that passesNone.calculate_dataframeis unchanged: it still returns a weightedMicroDataFramewith enum names, and still builds weights once per column, which it discards, plus once for the frame (4 for 3 columns in the new tests' fixture, against 33 on master). Removing the per-column weights without changing its enum decoding would need a separate way to turn weights off, which this PR does not add.Microsimulationused inside a formula of a plainSimulation(not aMicrosimulation) still gets weighted results, as on master: only a runningMicrosimulationcall marks a call as internal. No consumer does this.Tests
New
tests/core/test_microsimulation_internal_calls.py, with a fixture system intests/fixtures/microsimulation_internal_calls.py:calculate, fast-cache re-read,calculate_add, year and monthcalculate_divide), trace off and on; a branch, trace off and on; a reform's baseline; aMicrosimulationbuilt in aMicrosimulation's formula and in a plainSimulation's formula; a worker thread with and without a copied context; a context copied in a formula and used after the calculation ends; a call in a copied context that keeps running after the call it was copied from has ended; a context copied during a nested call and used while an enclosing call still runs; the fast-cache re-read; explicit arguments inside and at the top level;calculate,calculate_add,calculate_divide,calculate_outputandcalculate_dataframeare still weighted, and each top-level call builds exactly one weighted series (foradds/subtracts,defined_for, a monthly variable over a year, a yearly variable over a month, ADD and DIVIDE), withhousehold_weightcalculated once;RuntimeError, aBaseException, trace on), an ADD or DIVIDE term raises, a formula catches a branch's error, or a branch input set during a calculation makes core run it a second time, calls from outside a calculation are weighted again;Microsimulationstores and returns what aSimulationdoes: same dtype, bytes and writability for every stored array;Simulation, and top-level enumcalculate_add/calculate_dividedecode when asked.test_enum_read_with_divide_over_a_year_gives_member_names_in_a_microsimulationrecorded the #658 bug; it now checks that the codes match aSimulation.test_microsimulation_calculate_signatures.pychecks the newNonedefaults.Run: each new and modified test, selected individually by node ID (all pass; the thread and copied-context tests also with
-X thread_inherit_context=1), andruff format/ruff checkon the changed files. The rest of the suite was not run locally; CI runs it.🤖 Generated with Claude Code