Skip to content

Store one cliff result per simulation run in the v2 database, with grants - #4327

Draft
anth-volk wants to merge 4 commits into
masterfrom
v2plan/c1-37-cliff-results
Draft

anth-volk wants to merge 4 commits into
masterfrom
v2plan/c1-37-cliff-results

Conversation

@anth-volk

Copy link
Copy Markdown
Collaborator

Fixes #4326

Summary

Adds revision bc334f9877b7 to the v2 chain (on f519a77ba122, autogenerated with --rev-id). It creates cliff_impacts: one row per cliff calculation of a simulation run, with the calculation's state and, once it succeeds, the run's cliff gap and cliff share. It also declares the table's grants.

Design decisions

The simulation API's change that writes the table and its change that starts calculations agreed this design before the revision was built:

  • One result per run, linked to the run, not a report. Every report naming the run shares the result. Cliffs need marginal tax rates, which recalculate each household with extra earnings, so a shared run never stores the cliff columns, and US national runs keep their region-group split. A request asking for cliffs starts a separate calculation of the run.
  • The row records the calculation's state: status (the v2_simulation_status type runs use), modal_call_id, error_message (code: message), started_at and completed_at. This lets a request tell "no result" from "being calculated" from "done".
  • At most one calculation per run that hasn't failed (uq_cliff_impacts_unfailed, partial unique on simulation_run_id where status <> 'failed', also declared for SQLite).
    • The simulation service inserts with ON CONFLICT (simulation_run_id) WHERE status <> 'failed' DO NOTHING, so two requests at once start one calculation.
    • A failed calculation never counts, as for runs: it stays as a record, and the next request inserts another.
    • Rejected: one row per run that a retry resets. That would need wider grants, and a stale call could write onto the reset row.
  • Checks: a succeeded or failed row has completed_at; a succeeded row has both numbers, and no other row has either. The numbers aren't range-checked: .py doesn't keep the share to at most 1, and a filtered simulation with no adults gives NaN.
  • CASCADE on the run, as poverty and inequality have. The result is part of its run, and nothing references it.
  • No version columns. A result inherits its run's catalog and package versions, so it is calculated only for a run on the versions the simulation API serves.
  • An index on (status, started_at), so the simulation API's stuck-run sweep can fail a calculation whose worker ended without an outcome.

Grants (policyengine_api/data/v2/privileges.py)

Role Privileges on cliff_impacts
policyengine_v2_simulation_service SELECT, INSERT, UPDATE (status, modal_call_id, error_message, completed_at, updated_at): insert a calculation, record its Modal call, or fail it when Modal refuses the call
policyengine_v2_modal_worker SELECT, UPDATE (status, modal_call_id, error_message, started_at, completed_at, updated_at, cliff_gap, cliff_share): claim, finish or fail; no INSERT
policyengine_v2_runtime SELECT
policyengine_v2_catalog_writer, policyengine_stage12_<environment> none

Neither simulation API role may change the run a row names or delete a row. The deploy's existing reconcile_v2_privileges.py apply and verify steps grant and check these after the schema upgrade; no role needs creating.

The revision

  • Post-generation correction: status reuses the baseline's v2_simulation_status type (postgresql.ENUM(..., create_type=False)), as 1356f658347d does for simulation_runs.status. The generated create_table would create the type again and fail. This is the only correction, and it is commented in the revision.
  • Downgrade: drops only the table, with its rows, and leaves the type to simulation_runs.
  • Compatibility: it writes no data and changes no other table. The previous Public API revision doesn't map the table, so it keeps working at this revision.
  • Head: V2_ALEMBIC_HEAD_REVISION moves to bc334f9877b7.
  • Out of scope here:
    • report_runs' four cliff columns stay until a later revision drops them.
    • The Public API doesn't read the table yet.
    • Until the simulation API writes the table, no run has a cliff calculation.
  • Another revision on the same parent: Store dynamics like policies, create them through /v2/dynamics, and keep gov.simulation parameters out of policies #4325 also adds one on f519a77ba122. Whichever merges second merges master and re-points its down_revision.

Docs

  • docs/engineering/skills/alembic-migrations.md: a section for this revision, plus the two simulation roles' rows in the role table.
  • The downgrade chains in that file, docs/migration/stage-9-v2-metadata.md and docs/migration/stage-12a-user-login.md.

Testing

Test-first: the first commit holds the failing tests, seen failing because there was no CliffImpact model, no cliff_impacts grants and no revision bc334f9877b7. The tests below were run by node ID against a disposable PostgreSQL 17 database.

  • Unit:
    • test_cliff_impact_models covers the columns, checks and indexes, and checks that SQLite keeps one unfailed calculation per run and refuses numbers on any row but a succeeded one.
    • The privilege manifest's tests for both simulation roles, the runtime role and the new table.
    • The revision chain, and this revision's operations and its one correction.
    • The v2 lifecycle script's test list.
    • test_simulation_run_models' per-simulation result tables.
  • Integration (test_alembic_v2_cliff_impacts, PostgreSQL):
    • the revision adds only its table and no enum type;
    • the downgrade leaves the type, and upgrading again works;
    • two inserts at once start one calculation, and a failed one is retried;
    • the checks, and the CASCADE.
  • Integration (test_v2_simulation_role_privileges, as the managed roles):
    • the simulation service inserts, joins, records a call and fails a calculation, and is refused claiming, writing numbers, changing the run or deleting;
    • the Modal worker claims and finishes, and is refused inserting, changing the run or deleting;
    • the runtime role reads a report run's results and is refused writing;
    • the catalog writer and the Stage 12 role can't read;
    • each simulation role can update exactly its declared columns.
  • Integration (test_alembic_v2_simulation_runs): the head foreign-key test now counts cliff_impacts among the simulation side's tables.
  • Checks:
    • alembic -c alembic-v2.ini check: no drift;
    • reconcile_v2_privileges.py apply --create-missing-roles and verify: pass;
    • also passing: lint-imports, python -m scripts.rules validate, python scripts/run_quality_guards.py, ruff, ruff format and mypy.
  • Explicit mypy on the changed policyengine_api/data/v2/models files reports the Field(sa_type=...) overload error on the new status, started_at and completed_at fields. That is the same error the existing SimulationRun fields have on master; the directory is outside make typecheck-v2.
  • Environment variables: the one CI script change adds a test file to .github/scripts/test_alembic_v2_lifecycle.sh.
    • Its tests read V2_MIGRATION_DATABASE_URL and V2_ALEMBIC_DISPOSABLE_TEST. alembic-v2-check.yml sets both as literal job-level env: values, for the job's own PostgreSQL service.
    • The deploy steps that apply the revision and grants are unchanged: seed-v2-database.yml, with V2_MIGRATION_DATABASE_URL from each GitHub environment's secrets.
  • No inline suppressions added.

🤖 Generated with Claude Code

anth-volk and others added 3 commits November 17, 2026 17:36
Each simulation run can have one cliff calculation that hasn't failed,
holding its state and, once it succeeds, the run's cliff gap and cliff
share. The tests cover the model, its checks and partial unique index,
the revision that adds the table, and the grants: the simulation service
starts calculations, the Modal workers finish them, the Public API reads
them, and the catalog writer and the Stage 12 role can do neither.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add cliff_impacts (revision bc334f9877b7, autogenerated on f519a77ba122):
one row per cliff calculation of a simulation run, with its state and,
once it succeeds, the run's cliff gap and cliff share. A partial unique
index allows one calculation per run that hasn't failed, so identical
requests join it and a failed one never counts. The row is deleted with
its run, and inherits the run's versions. Its status reuses
v2_simulation_status, the revision's one post-generation correction.

The simulation service may insert calculations and record or fail their
Modal calls, the Modal workers may write their state and results, the
Public API may read them, and no other role may touch them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Stage 9 and Stage 12A runbooks' downgrade chains now start with
bc334f9877b7, and the head foreign-key test counts cliff_impacts among
the simulation side's tables. The changelog fragment and the skill
section say what each role may do, since the simulation API writes the
table only once its own change lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.67%. Comparing base (aa79db7) to head (a4be631).
⚠️ Report is 4 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4327      +/-   ##
==========================================
- Coverage   93.69%   93.67%   -0.03%     
==========================================
  Files         397      397              
  Lines       22885    22900      +15     
  Branches     3599     3599              
==========================================
+ Hits        21443    21451       +8     
- Misses        839      846       +7     
  Partials      603      603              
Flag Coverage Δ
integration 57.72% <100.00%> (+0.02%) ⬆️
unit 90.95% <100.00%> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The v2 metadata now holds 51 tables.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

1 inactive deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Store one cliff result per simulation run in the v2 database, with grants

1 participant