Skip to content
Draft
1 change: 1 addition & 0 deletions changelog.d/v2-dynamics-like-policies.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Store dynamics, a simulation's model assumptions, as policies are stored: once per country, catalog version and content hash, with no name or description. `POST /v2/dynamics` finds or creates one (201 or 200) from parameters under `gov.simulation`, and in the US also `simulation`, refusing other parameters and an empty dynamic with 422, and `GET /v2/dynamics/{dynamic_id}` reads one in its country. `POST /v2/policies` refuses those parameters with 422 (code `dynamic_parameters`, naming them). `/v2/simulations` and `/v2/reports` refuse a dynamic the country does not store (404) or one that sets parameters the running catalog lacks (422 `missing_parameters`). The v1-to-v2 policy copy splits a v1 policy's `gov.simulation` and US `simulation` values into a dynamic, rewriting the UK's old `labor_supply_responses` spelling, and maps each v1 policy to its policy and dynamic (`legacy_policy_mappings.dynamic_id`).
16 changes: 13 additions & 3 deletions docs/engineering/migration-contracts.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ Generated from `policyengine_api/migration_registry.py` and `tests/contract/regi
| Metric | Count |
| --- | ---: |
| route group count | 9 |
| workflow count | 18 |
| request count | 101 |
| workflow count | 19 |
| request count | 103 |
| db entity count | 6 |
| sim flow count | 3 |

Expand All @@ -19,7 +19,7 @@ Generated from `policyengine_api/migration_registry.py` and `tests/contract/regi
| `health` | `health`, `simulation-gateway-check`, `liveness-check`, `readiness-check`, `deployment` | `none` | `none` |
| `specification` | `specification` | `none` | `none` |
| `metadata` | `metadata`, `countries`, `datasets`, `economy-options`, `model-catalog`, `parameter-values`, `parameters`, `regions`, `tax-benefit-model-versions`, `tax-benefit-models`, `variables` | `metadata` | `none` |
| `policy` | `policy`, `policies`, `user-policy`, `user-policies` | `policy` | `none` |
| `policy` | `policy`, `policies`, `dynamics`, `user-policy`, `user-policies` | `policy` | `none` |
| `household` | `household`, `households`, `household-calculations`, `user-household`, `user-households`, `calculate`, `calculate-full` | `household` | `household` |
| `economy` | `economy` | `simulation` | `economy` |
| `simulation` | `simulation`, `simulations`, `user-simulations` | `simulation` | `economy` |
Expand Down Expand Up @@ -49,6 +49,16 @@ Generated from `policyengine_api/migration_registry.py` and `tests/contract/regi
| `GET` | `/v2/policies/{policy_id}?country_id=us` | 200 | `policy` | none | `status`, `message`, `result.item.id`, `result.item.country_id`, `result.item.tax_benefit_model_id`, `result.item.parameter_values`, `result.item.created_at`, `result.item.updated_at`, `result.item.missing_parameters` |
| `GET` | `/v2/policies?country_id=us` | 200 | `policy` | none | `status`, `message`, `result.items`, `result.offset`, `result.limit`, `result.has_more` |

### `dynamic_resources_v2`

- Current contract: `typed_v2_resources`
- Future owner: Dynamics stored like policies

| Method | Path | Status | Route group | Required headers | Stable response fields |
| --- | --- | ---: | --- | --- | --- |
| `POST` | `/v2/dynamics?country_id=us` | 201 | `policy` | none | `status`, `message`, `result.item.id`, `result.item.country_id`, `result.item.tax_benefit_model_id`, `result.item.parameter_values`, `result.item.created_at`, `result.item.updated_at`, `result.item.missing_parameters` |
| `GET` | `/v2/dynamics/{dynamic_id}?country_id=us` | 200 | `policy` | none | `status`, `message`, `result.item.id`, `result.item.country_id`, `result.item.tax_benefit_model_id`, `result.item.parameter_values`, `result.item.created_at`, `result.item.updated_at`, `result.item.missing_parameters` |

### `user_policy_associations_v2`

- Current contract: `typed_v2_resources`
Expand Down
65 changes: 63 additions & 2 deletions docs/engineering/skills/alembic-migrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -310,8 +310,9 @@ production, so that its absence fails the deploy again.

The declarations include write privileges for code paths production doesn't
turn on: the dual-write copies into `legacy_policy_mappings` and
`legacy_household_mappings`, and the writes behind the native `/v2/policies`
and `/v2/households` routes. Narrow them when those tables are removed.
`legacy_household_mappings`, and the writes behind the native `/v2/policies`,
`/v2/dynamics` and `/v2/households` routes. Narrow them when those tables are
removed.

When a pull request adds or changes a v2 table, or a connector's use of one:

Expand Down Expand Up @@ -456,6 +457,8 @@ it publishes that version, with nothing to restore it. Downgrading past the
revision is therefore safe only while no version is pruned (no
`tax_benefit_model_versions` row has `catalog_values_pruned_at` set).
From the head, such a downgrade first runs the later revisions' downgrades:
`9d1d12a7ac58`'s needs `dynamics` empty (see "API v2 dynamics stored like
policies"),
`f519a77ba122`'s drops `user_invitations` and needs no recorded change with
source `invitation` (see "API v2 user invitations"),
`94dc43079186`'s drops every user's report-run record, the simulation and
Expand Down Expand Up @@ -1012,3 +1015,61 @@ change's source is `invitation`.
`test_user_invitations_revision_adds_invitations_and_dismissals_only` and
`test_user_invitations_revision_records_invitation_changes_until_downgraded`
in `tests/integration/test_alembic_v2_user_types.py` check both.

### API v2 dynamics stored like policies

Revision `9d1d12a7ac58` (parent `f519a77ba122`) was autogenerated, with no
post-generation corrections apart from `ruff format`, after these model
changes, so that dynamics are stored as policies are:

- `dynamics` drops `name` and `description`, which nothing wrote or read: a
user's saved dynamic will hold its name, as `user_policies` holds a
policy's. It gains `policies`' columns: `country_id` (a key to
`countries`), `tax_benefit_model_id` and `tax_benefit_model_version_id`
(indexed), each `RESTRICT`, and `canonicalization_version` and
`content_hash`, unique together
(`uq_dynamics_canonicalization_content_hash`), with the checks policies
have (a positive version, a 64-character hash). `(id, country_id)` is
unique (`uq_dynamics_id_country`), so other tables can name a dynamic of
one country.
- `parameter_values` gains `uq_parameter_values_dynamic_parameter_start_date`,
a partial unique index on `(dynamic_id, parameter_id, start_date)` for rows
with a dynamic: a dynamic sets one value per parameter and start date, as a
policy does. Rows without a dynamic never conflict in a unique key, so the
partial index refuses exactly what a full unique constraint would, but it
holds only dynamics' rows instead of every current-law value. Building it
still reads every row of `parameter_values` (millions of current-law
values), and, as Alembic creates it without `CONCURRENTLY`, inserts into
that table wait until the build finishes. It is also declared for SQLite,
so the unit tests' databases keep the same rule.
- `legacy_policy_mappings` gains the nullable `dynamic_id`, the dynamic the
v1-to-v2 policy copy split from a v1 policy: a key with the mapping's
country to `dynamics` (`fk_legacy_policy_mappings_dynamic_country`,
`RESTRICT`), indexed. A mapping without a dynamic is unaffected.

A dynamic is hashed by the policies' function (`canonical_policy_document`)
with the same canonicalization version, so identical content in a policy
and a dynamic has the same hash; they are in different tables. The runtime
role may now read and insert `dynamics`; the simulation service and its
Modal workers keep reading them. Which parameters a dynamic holds, and how
the copy splits a v1 policy, is in `policyengine_api/dynamic_parameters.py`.

Nothing wrote `dynamics` before this revision, and its new columns are
required with no server default, so the upgrade fails, changing nothing,
while `dynamics` has a row. Before deploying it to an environment, count
them read-only:

```sql
SELECT count(*) FROM dynamics;
```

The previous Public API revision maps `name` and `description` but never
reads or writes `dynamics`, and its copy writes mappings without
`dynamic_id`, which stays null, so it keeps working at this revision. The
simulation service selects only `dynamics.id`. The downgrade drops the new
columns, constraints and indexes and the mappings' `dynamic_id` with its
values, and adds `name` back as required, so it fails while `dynamics` has a
row. `tests/integration/test_alembic_v2_dynamics.py` checks that nothing else
changes, the downgrade and upgrade, both refusals, and the keys and unique
rules.

48 changes: 46 additions & 2 deletions docs/generated/migration_contracts.json
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
{
"metadata": {
"db_entity_count": 6,
"request_count": 101,
"request_count": 103,
"route_group_count": 9,
"sim_flow_count": 3,
"workflow_count": 18
"workflow_count": 19
},
"route_groups": [
{
Expand Down Expand Up @@ -51,6 +51,7 @@
"path_segments": [
"policy",
"policies",
"dynamics",
"user-policy",
"user-policies"
],
Expand Down Expand Up @@ -201,6 +202,49 @@
}
]
},
{
"current_contract": "typed_v2_resources",
"future_owner_pr": "Dynamics stored like policies",
"name": "dynamic_resources_v2",
"requests": [
{
"expected_status": 201,
"headers": [],
"method": "POST",
"path": "/v2/dynamics?country_id=us",
"route_group": "policy",
"stable_response_fields": [
"status",
"message",
"result.item.id",
"result.item.country_id",
"result.item.tax_benefit_model_id",
"result.item.parameter_values",
"result.item.created_at",
"result.item.updated_at",
"result.item.missing_parameters"
]
},
{
"expected_status": 200,
"headers": [],
"method": "GET",
"path": "/v2/dynamics/{dynamic_id}?country_id=us",
"route_group": "policy",
"stable_response_fields": [
"status",
"message",
"result.item.id",
"result.item.country_id",
"result.item.tax_benefit_model_id",
"result.item.parameter_values",
"result.item.created_at",
"result.item.updated_at",
"result.item.missing_parameters"
]
}
]
},
{
"current_contract": "typed_v2_resources",
"future_owner_pr": "PR 12A: User Login and v1-Anchored User Associations",
Expand Down
27 changes: 24 additions & 3 deletions docs/migration/stage-10-v2-policies.md
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,26 @@ the running catalog whatever country package version the v1 row was saved
under (its stored `api_version`), the same way the Public API simulates an old
policy with the running country package. A policy is copied when every
parameter path it sets exists in the running catalog and its periods and values
are valid. A saved-policy mutation commits
are valid.

The copy splits off the v1 policy's model assumptions: every value under
`gov.simulation.`, and in the US also `simulation.`
(`policyengine_api/dynamic_parameters.py`). It stores them as a v2 dynamic,
hashed and deduplicated as a policy is, and keeps the rest in the v2 policy,
which is empty when the v1 policy set nothing else. Before looking up the
parameters it rewrites the UK's old spelling
`gov.simulation.labor_supply_responses.` to
`gov.simulation.labour_supply_responses.`, the only one the UK catalog has; a
v1 policy that sets one parameter under both spellings with different values
fails as `conflicting_values`. The mapping in `legacy_policy_mappings` names the
policy and the dynamic (`dynamic_id`), or no dynamic when the v1 policy sets no
assumptions. A v1 policy copied before the split existed keeps its copy: its
assumptions stay policy values and its mapping names no dynamic, until the
one-time migration splits it again. Native `POST /v2/policies` refuses such
values with 422 instead (code `dynamic_parameters`), and they are created
through `POST /v2/dynamics`.

A saved-policy mutation commits
its source row, incremented revision, and complete event in one Cloud SQL
transaction. The same request then processes that source's pending events in
revision order and records `processed_at` only after the corresponding
Expand Down Expand Up @@ -206,14 +225,16 @@ Monitor structured events with metric names `v1_policy_mirror_operations` and
| `failure_category` | `failure_reason` |
| --- | --- |
| `catalog_or_translation` | `missing_parameters` (the policy sets a parameter the running catalog lacks), `invalid_parameter_values`, `empty_parameter_values`, `invalid_period`, `conflicting_values`, `too_many_values` (more than 1,000 dated values), `catalog_absent` (no catalog for the running PolicyEngine.py version), `catalog_query_failed`, `catalog_version_not_found`, `catalog_mismatch` |
| `integrity` | `mapping_conflict`, `content_hash_collision`, `stored_policy_integrity`, `saved_policy_integrity` (any integrity check of a saved-policy copy) |
| `integrity` | `mapping_conflict`, `content_hash_collision` (of the policy or the dynamic), `stored_policy_integrity`, `stored_dynamic_integrity`, `saved_policy_integrity` (any integrity check of a saved-policy copy) |
| `database` | `database_error` |
| `configuration` | `configuration` |
| `unexpected` | `missing_reform_policy` (a saved policy's v1 reform could not be loaded), `unexpected` |
| `missing_mirror_snapshot`, `missing_mirror_revision` | the same value as the category |

Events include the legacy integer ID,
destination UUID when committed, attempted and completed database sources, and
destination UUID when committed (a policy copy's record also names its
dynamic's, `destination_dynamic_id`, and whether it created it,
`dynamic_created`), attempted and completed database sources, and
duration. They do not include policy JSON, presentation data, database URLs, or
credentials.

Expand Down
Loading
Loading