Skip to content

Store dynamics like policies, create them through /v2/dynamics, and keep gov.simulation parameters out of policies - #4325

Draft
anth-volk wants to merge 8 commits into
masterfrom
v2plan/c1-36-dynamics
Draft

anth-volk wants to merge 8 commits into
masterfrom
v2plan/c1-36-dynamics

Conversation

@anth-volk

Copy link
Copy Markdown
Collaborator

Fixes #4324

Summary

Dynamics hold a simulation's model assumptions: every parameter under gov.simulation., and in the US also the top-level simulation. tree. This stores them as policies are stored, creates them through /v2/dynamics, and keeps them out of policies.

  • Revision 9d1d12a7ac58 (parent f519a77ba122), autogenerated, with no corrections apart from ruff format:

    • dynamics gains policies' columns: country_id, tax_benefit_model_id and tax_benefit_model_version_id (all RESTRICT), plus canonicalization_version and content_hash, unique together. It drops name and description, which nothing wrote or read.
    • parameter_values gains a partial unique index on (dynamic_id, parameter_id, start_date) for rows that have a dynamic.
    • legacy_policy_mappings gains a nullable dynamic_id, a key (with the mapping's country) to dynamics.
    • The runtime role may read and insert dynamics.
  • policyengine_api/dynamic_parameters.py:

    • which parameters a dynamic holds, per country;
    • the UK rewrite of gov.simulation.labor_supply_responses. to labour_supply_responses. (the US name stays);
    • split_legacy_policy_json, which the one-time migration will reuse.
  • POST /v2/dynamics (country_id, optional policyengine_version) binds the values to a catalog version and stores them once per content hash, with the policies' canonical document and canonicalization version. It answers:

    • 201 when it creates a dynamic, 200 when identical content is already stored;
    • 422 with code policy_parameters, naming them, for parameters that aren't model assumptions;
    • the usual request-validation 422 for an empty dynamic: for no assumptions, send dynamic_id null.
  • GET /v2/dynamics/{dynamic_id} reads one in its country, with missing_parameters. There is no list route.

  • POST /v2/policies refuses model assumptions with 422, code dynamic_parameters, naming them.

  • /v2/simulations and /v2/reports (economy reports and budget windows) check each named dynamic after the policies:

    • one the country doesn't store answers 404;
    • one that sets parameters the running catalog lacks answers 422 missing_parameters.

    Re-runs leave a dynamic's missing parameters to the simulation service, which refuses them.

  • The v1-to-v2 policy copy splits a v1 policy's model assumptions into a dynamic, rewriting the UK's old spelling first, and keeps the rest in the policy, which is empty when nothing else remains. Its mapping records both. Copies made before this keep their mapping and have no dynamic until the one-time migration splits them. The copy's log record gains destination_dynamic_id and dynamic_created, and the failure reason stored_dynamic_integrity.

The PR adds no inline suppression (noqa or type: ignore).

Deploy order

  • The simulation API: its deployed code declares dynamics with id alone and selects only dynamics.id, so dropping name and description breaks none of its reads. Its test fixture that inserts a dynamic with a name runs against the Public API commit it pins. Its C1 · 44 moves that pin to this PR's merge and updates the fixture.

  • Existing rows: nothing wrote dynamics before this, and the new columns are required without a default, so the upgrade fails, changing nothing, while dynamics has a row. Check each environment first with SELECT count(*) FROM dynamics;. The downgrade adds name back as required, so it fails while dynamics has a row.

  • The index build: it reads all of parameter_values and holds back inserts into it until it finishes.

  • The previous Public API release never reads or writes dynamics, so it keeps working at this revision.

  • Variables: the migration and grant steps (seed-v2-database.yml, from push.yml with secrets: inherit) read only:

    • V2_MIGRATION_DATABASE_URL and V2_DATA_WRITE_DATABASE_URL, GitHub environment secrets in staging and production;
    • V2_SUPABASE_PROJECT_REF and V2_SUPABASE_ENVIRONMENT, environment variables in both;
    • PRODUCTION_V2_SUPABASE_PROJECT_REF, a repository variable;
    • DEPLOYMENT_ENVIRONMENT, the workflow input.

    All exist; this PR adds none.

Other open pull requests

#4029 already conflicts with master. This PR adds text conflicts with it only in tests/contract/registry.py and tests/contract/test_app_v2_workflow_contracts.py, where both add workflow entries. It doesn't change the v1 policy routes that #4029 changes.

Testing

  • Tests run, by node ID only (new and changed tests): the unit tests, and Postgres 17 integration tests run as the runtime role:
    • test_alembic_v2_dynamics.py: the revision changes nothing else, upgrades and downgrades exactly, refuses both ways with a stored dynamic, and keeps its keys and unique rules;
    • test_v2_dynamic_persistence.py: dynamic dedupe and reads, and the copy's split and UK rewrite;
    • one creation test of each submission route.
  • Checks: ruff, ruff format, mypy (with the changed data/v2 files), lint-imports, python -m scripts.rules validate, scripts/run_quality_guards.py, alembic -c alembic-v2.ini check, and reconcile_v2_privileges.py apply --create-missing-roles then verify on a disposable database.
  • Generated docs: scripts/export_migration_contracts.py regenerated the migration contract docs, with the new dynamic_resources_v2 workflow.

🤖 Generated with Claude Code

anth-volk and others added 7 commits November 17, 2026 15:27
…ing)

Dynamics get policies' country, catalog version and content-hash columns,
lose name and description, and a dynamic's values are unique per parameter
and start date. legacy_policy_mappings names an optional dynamic of the same
country. A new pure module says which parameters a dynamic holds and splits a
v1 policy's values, rewriting the UK's old labour-supply spelling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Revision 9d1d12a7ac58 (on f519a77ba122, autogenerated with no corrections
apart from ruff format) gives dynamics policies' country, tax-benefit model,
catalog version, canonicalization version and content hash, unique per
version and hash, and drops name and description, which nothing wrote or
read. parameter_values gains a partial unique index on a dynamic's values per
parameter and start date, and legacy_policy_mappings a nullable dynamic_id,
a RESTRICT key to a dynamic of the mapping's country. The runtime role may
read and insert dynamics.

policyengine_api.dynamic_parameters says which parameters a dynamic holds
(every gov.simulation. parameter, plus the US simulation. tree), rewrites
the UK's old labor_supply_responses spelling, and splits a v1 policy's values
into the policy's and the dynamic's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…plit copy (failing)

POST /v2/dynamics finds or creates a dynamic of gov.simulation parameters
(201 or 200), refuses an empty one and one with other parameters (422), and
GET /v2/dynamics/{id} reads one in its country. POST /v2/policies refuses
gov.simulation and US simulation parameters with 422, naming them.
/v2/simulations and /v2/reports refuse a dynamic not stored in the country
(404) or setting parameters the running catalog lacks (422). The v1-to-v2
copy splits a v1 policy's model assumptions into a dynamic, rewrites the UK's
old spelling, maps the v1 policy to both, and logs the dynamic.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lit the copy

POST /v2/dynamics binds model assumptions (parameters under gov.simulation,
and in the US simulation) to a catalog version and stores them once per
content hash with the policies' canonical document, answering 201 or 200;
GET /v2/dynamics/{dynamic_id} reads one in its country with its
missing_parameters. A dynamic setting any other parameter answers 422 with
code policy_parameters, and an empty one fails request validation. The
dynamic service reports database failures as DynamicStorageError (503).

POST /v2/policies answers 422 with code dynamic_parameters for a policy that
sets model assumptions. Economy simulations, economy reports and budget
windows refuse a dynamic the country does not store (404) or one setting
parameters the running catalog lacks (422 missing_parameters).

The v1-to-v2 policy copy splits a v1 policy's model assumptions into a
dynamic, rewriting the UK's old labor_supply_responses spelling first, and
records the dynamic in legacy_policy_mappings.dynamic_id; its log record names
the dynamic, and classifies the dynamic's integrity failures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	policyengine_api/services/v2/simulation_service/services.py
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dynamic gains Policy's relationships to its tax-benefit model and catalog
version, so the ORM stores a new catalog version before a dynamic in one
flush, and the unused relationships between dynamics and legacy policy
mappings are gone, so mapper configuration emits no SAWarning again. Tests
cover both, the copy losing the mapping race after creating a dynamic, and a
dynamic that cannot be written. The mapping input's country is a CountryId.
Docstrings and descriptions name the dynamic errors, the re-run description
says that the simulation service refuses a dynamic's missing parameters, and
the migration skill says that building the new index reads all of
parameter_values and holds back inserts until it finishes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 17, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 64.37055% with 150 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.21%. Comparing base (aa79db7) to head (23d17c6).

Files with missing lines Patch % Lines
policyengine_api/services/v2/dynamics/services.py 33.33% 52 Missing ⚠️
...icyengine_api/fastapi_routes/v2/dynamics/routes.py 59.64% 22 Missing and 1 partial ⚠️
policyengine_api/dynamic_parameters.py 62.85% 10 Missing and 3 partials ⚠️
...olicyengine_api/services/v2/dynamics/validators.py 62.96% 10 Missing ⚠️
...engine_api/services/v2/dynamics/transformations.py 50.00% 9 Missing ⚠️
...icyengine_api/fastapi_routes/v2/policies/routes.py 12.50% 7 Missing ⚠️
.../services/v2/dynamics/database_connectors/reads.py 57.14% 6 Missing ⚠️
...engine_api/services/v2/policies/transformations.py 68.42% 5 Missing and 1 partial ⚠️
...olicyengine_api/services/v2/policies/validators.py 45.45% 4 Missing and 2 partials ⚠️
...ervices/v2/dynamics/database_connectors/creates.py 61.53% 5 Missing ⚠️
... and 5 more

❌ Your patch check has failed because the patch coverage (64.37%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4325      +/-   ##
==========================================
- Coverage   93.69%   93.21%   -0.49%     
==========================================
  Files         397      408      +11     
  Lines       22885    23282     +397     
  Branches     3599     3648      +49     
==========================================
+ Hits        21443    21703     +260     
- Misses        839      978     +139     
+ Partials      603      601       -2     
Flag Coverage Δ
integration 57.80% <64.37%> (+0.10%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch was previously deployed

1 inactive deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Store dynamics like policies, create them through /v2/dynamics, and keep gov.simulation parameters out of policies

1 participant