Skip to content

Read report cliffs from simulation runs' cliff results, and take include_cliffs on re-runs - #4329

Draft
anth-volk wants to merge 7 commits into
v2plan/c1-37-cliff-resultsfrom
v2plan/c1-40-report-cliffs
Draft

anth-volk wants to merge 7 commits into
v2plan/c1-37-cliff-resultsfrom
v2plan/c1-40-report-cliffs

Conversation

@anth-volk

@anth-volk anth-volk commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #4328

Stacks on #4327 (v2plan/c1-37-cliff-results, revision bc334f9877b7, table cliff_impacts); retarget to master once #4327 merges.

Summary

  • Cliffs come from the simulation runs. An economy report run's result.cliff_impact is its two simulation runs' succeeded cliff_impacts rows, {baseline, reform}, once both have one, and null otherwise. Nothing in the Public API reads report_runs' four cliff columns any more, so a later revision can drop them. A report run that stored cliffs only in those columns now reads none.

  • Each economy report read gives cliffs: how far the two cliff calculations have got, for the run read in any status. It appears on the get, and on the create and re-run answers, but not on lists or searches, and is null for a report with no run.

    "cliffs": {
      "status": "running",
      "baseline": {"status": "succeeded", "error_message": null},
      "reform":   {"status": "running",   "error_message": null}
    }
    • A side's status: not_calculated, pending, running, succeeded, or failed. A side is failed when its newest calculation failed and none is under way; error_message then says why.
    • The overall status: succeeded once both sides are. Otherwise it is failed if either side is, else not_calculated if either side is, else running if either side is, else pending.
    • Which row a side reads: its simulation run's calculation that hasn't failed (at most one, by uq_cliff_impacts_unfailed), else its newest failed one. One query per distinct simulation run, so a report naming one run on both sides reads it once.
  • include_cliffs on re-runs. POST /v2/reports/{report_id}/runs takes include_cliffs (default false).

    • The Public API sends "include_cliffs": true to the simulation service only when asked, so a re-run that doesn't ask still works on a simulation service whose re-run body takes only force.
    • A household report or budget window asking for cliffs answers 422 before the simulation service is asked or anything is recorded; their creates already refuse the field.
    • The economy create already sends include_cliffs and is unchanged.
  • File split. stored_reports.services reached its 800-line limit, so the code that reads reports as the requesting user sees them moves unchanged to stored_reports.requester_views.services. _requesters_view is renamed requesters_view there, because another module now imports it.

  • Changelog fragment: changelog.d/report-cliffs-from-simulation-runs.changed.md.

Merge and deploy order

  1. Store one cliff result per simulation run in the v2 database, with grants #4327's revision bc334f9877b7, applied with its grants in staging and production. This PR reads cliff_impacts as the runtime role.
  2. The simulation service's C1 · 38 (Calculate each simulation run's cliff gap and share as its own cliff result policyengine-sim-api#1096) as the default worker's v2 app. It calculates and writes each run's cliff result, and stops writing the four columns.
  3. The simulation service's C1 · 39 (Start each side's cliff calculation when a report asks for cliffs policyengine-sim-api#1099). It starts the cliff calculations on creates and re-runs with include_cliffs.

The read is safe to deploy after step 1 alone. Until steps 2 and 3 deploy, a caller's include_cliffs: true answers 422:

  • on a create, the simulation service's own refusal;
  • on a re-run, its validation error naming include_cliffs.

The Public API passes either on as 422.

Migration notes

  • Route group: v2 reports (/v2/reports: read, create, re-run). Nothing on the Flask path changes.
  • API contract changes, intended:
    • economy report items gain cliffs;
    • the re-run body gains include_cliffs;
    • result.cliff_impact now comes from the simulation runs' cliff results.
  • Generated documents: scripts/export_migration_contracts.py regenerates both migration contract documents unchanged; they don't record this route's answer shape.
  • Deployment: no deploy step, workflow, image, environment variable or secret changes.

Inline suppressions added

  • # noqa: PLR0913 on read_economy_result_rows, the database read of a report run's result rows. It now takes the cliff calculations the report service has already read, so a run's cliffs are read once whatever its status. Its other inputs (the session, the country, the run, its reform simulation run and the local-area flag) are each needed, and a parameter object used only by this one call would add a type without simplifying anything.
  • # noqa: PLR0913 on the integration test helper store_cliff_calculation, which takes the connection, the simulation run, the status and each column a test sets.

Testing

Each test was written first and seen failing for the expected reason, and committed with "(failing)" in the subject before the code that passes it. I ran only the tests this PR adds or changes, by node ID: 35 node IDs, 78 cases with their parameters, all passing. The PostgreSQL integration tests ran against a disposable PostgreSQL 17 database built from this branch, with reconcile_v2_privileges.py apply --create-missing-roles and verify, as the runtime role.

  • New:
    • tests/unit/v2/test_report_cliff_reads.py: each side's row; the side and overall statuses; cliff_impact; one run on both sides;
    • tests/integration/test_v2_report_cliff_reads.py: succeeded, still calculating and failed sides; the four columns not read; no run;
    • route, OpenAPI, client-body and re-run tests in test_report_read_routes.py, test_report_run_routes.py, test_simulation_report_reruns.py, test_v2_report_runs.py and test_v2_report_creation.py.
  • Changed:
    • tests/fixtures/economy_results.py stores a result's cliffs as each simulation run's succeeded cliff_impacts row instead of the report run's columns;
    • the golden rebuild test expects no cliff impact from the report-run writer's rows.

Also run: ruff, ruff format, make typecheck-v2 (mypy), lint-imports, python -m scripts.rules validate, python scripts/run_quality_guards.py and scripts/export_migration_contracts.py. No file under policyengine_api/data/v2/ changes, and there is no revision or grant change.

Reviewed in one adversarial round, which found no bugs. Its code-quality findings are fixed in this PR:

  • the economy read has its own method;
  • the descriptions of cliff_impact, the read and create routes, and re-runs of runs on older versions are corrected;
  • a golden test that couldn't fail is replaced by one that fails if the report run's result columns hold a cliff column again.

🤖 Generated with Claude Code

anth-volk and others added 7 commits November 17, 2026 19:22
…failing)

A report run's cliff impact comes from its two simulation runs' cliff_impacts
rows, not from report_runs' four cliff columns, and an economy report read
says how far each side's cliff calculation has got (cliffs). The fixtures
store a result's cliffs as each simulation run's succeeded calculation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An economy report run's cliff impact is its two simulation runs' succeeded
cliff_impacts rows, and no read uses report_runs' four cliff columns any
more, so C1 · 41 can drop them. Each side reads its run's calculation that
hasn't failed, else its newest failed one. Economy report reads (get, and
the create and re-run answers) gain cliffs: an overall status and each
side's status and error message, for a run in any status.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /v2/reports/{id}/runs takes include_cliffs and passes it to the
simulation service, sending it only when asked; a household report's or
budget window's re-run asking for cliffs is refused with 422 before the
service is asked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /v2/reports/{id}/runs takes include_cliffs and sends it to the
simulation service only when true, so a re-run that doesn't ask still runs
on a simulation service that doesn't take the field yet. A household
report's or budget window's re-run asking for cliffs answers 422 before the
service is asked or anything recorded. The create body's include_cliffs now
says what it starts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
stored_reports.services reached its 800-line limit. The code that reads
reports as the requesting user sees them (their saved report's label and
their newest requested run) moves unchanged to
stored_reports.requester_views.services; _requesters_view is renamed
requesters_view there, since the report service now imports it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 1's quality findings: the economy read moves into
_economy_report, as household reports and budget windows have theirs; the
result's cliff_impact, the read and create routes, and a re-run of a run on
older versions now say how cliffs read and start, and that a simulation
service that doesn't calculate cliffs per simulation run yet refuses
include_cliffs with 422; get_report names the reform check for a run in any
status. The golden test that couldn't fail is replaced by one that fails if
the report run's result columns hold a cliff column again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.31933% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.70%. Comparing base (a4be631) to head (6ef47cb).

Files with missing lines Patch % Lines
...ices/v2/stored_reports/requester_views/services.py 94.28% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@                      Coverage Diff                       @@
##           v2plan/c1-37-cliff-results    #4329      +/-   ##
==============================================================
+ Coverage                       93.67%   93.70%   +0.02%     
==============================================================
  Files                             397      398       +1     
  Lines                           22900    22972      +72     
  Branches                         3599     3608       +9     
==============================================================
+ Hits                            21451    21525      +74     
+ Misses                            846      845       -1     
+ Partials                          603      602       -1     
Flag Coverage Δ
integration 57.83% <94.95%> (+0.11%) ⬆️
unit 90.94% <71.42%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

This branch was successfully deployed

1 active deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant