Skip to content

Calculate each simulation run's cliff gap and share as its own cliff result - #1096

Draft
anth-volk wants to merge 19 commits into
mainfrom
v2plan/c1-38-run-cliff-results
Draft

anth-volk wants to merge 19 commits into
mainfrom
v2plan/c1-38-run-cliff-results

Conversation

@anth-volk

@anth-volk anth-volk commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1095

What this does

Each simulation run can now have its own cliff result: its cliff gap and cliff share, in a row of the Public API's cliff_impacts.

The cliff calculation

  • Where it runs. A call of the existing v2 simulation workers (run_single_simulation_us, run_single_simulation_uk) whose context names a cliff result (Stage12InvocationContext.cliff_result_id) calculates that run's cliffs instead of its output file. No new Modal function, image, secret or manifest entry is needed.
  • Claim. The worker claims the row under its Modal call ID, as a simulation run is claimed. A restarted call resumes. A second start, or a row another call holds or that has finished, calculates nothing and returns a deduplicated marker.
  • Input check. The payload must be the simulation of the row's run, on the run's year and versions; otherwise the row fails with cliff_result_input_mismatch. The calculation then checks the installed bundle, as for any run.
  • Calculation. It plans the simulation with the country model's default columns plus .py's cliff_gap, is_on_cliff and is_adult (resolve_cliff_output_plan), calculates it again from its dataset and policy, and records .py's weighted cliff gap and cliff share. It writes them, or the redacted failure, only while its call holds the row. A calculation with no adults fails with cliff_share_undefined.
  • US national runs are split into the 20 region groups whenever the run's own calculation would be (should_segment_cliff_calculation). Each person's cliff columns depend only on their own household, and the share is taken once from the merged rows. A test checks that 20 merged region groups give the national result.
  • Nothing else is written. The call doesn't touch the simulation run, its output file or any report run. A coordinator call that names a cliff result is refused.

Other changes

  • Stuck-run sweep. It fails a cliff result whose call ended, one left pending with no call, and one past the age limit (RunKind.CLIFF_RESULT).
  • Report run completion no longer sets or names report_runs' four cliff columns, so the Public API can drop them. The Modal worker role's privilege list no longer includes them.
  • Table model. Sim models cliff_impacts with the service's tables (cliff_impact_tables.py), with both roles' privileges as the Public API grants them. The model matches the table the Public API's revision builds.
  • Context compatibility. A context that names no cliff result leaves the field out of its JSON (exclude_if), so v2 applications deployed before this change still accept every context the service sends.
  • Rule proof fixtures. The missing-privileges rule's two grant fixtures (rules/fixtures/sim.persistence.missing-privileges-reported/) add the service's cliff_impacts grants, since they list every privilege the service needs. The bad grants still leave out the same two. Files under rules/ need a core developer's approval other than the author's (sim.process.rule-machinery-code-owned).
  • A move with no behaviour change. report_run_store.py's run statuses, kinds and outcomes move to run_states.py (re-exported), since the file was at its line limit.

Nothing starts a cliff calculation yet, and a report asking for cliffs is still refused (cliffs_not_available). Starting calculations from /reports and re-runs follows separately.

Must merge and deploy first

  1. The Public API's cliff_impacts revision (bc334f9877b7, PolicyEngine/policyengine-api branch v2plan/c1-37-cliff-results) must be merged and applied in staging and production.
    • The service's /ready privilege check now names cliff_impacts.
    • The stuck-run sweep lists the table on every pass, so a database without it fails the sweep for every kind of run.
  2. This PR then moves libs/policyengine-simulation-persistence/public-api-commit to that merge commit. Until then, the Public API schema job compares against the pinned commit 63cce8273, which has no cliff_impacts, so its comparison and write tests fail.

#1078 (one library of table models), which this was built on, has merged.

Deployment and environment variables

  • MODAL_WORKER_DATABASE_URL is the only setting the new code reads. report_run_access.cliff_result_store reads it, as report_run_store does.
    • Provider: the Modal secret policyengine-modal-worker-database.
    • Wiring: v2_app.py already attaches that secret to run_single_simulation_us and run_single_simulation_uk (report_run_writer_secrets), and run_sweep_app.py attaches it to the sweep. modal deploy checks its required key.
    • Unverified here: that the secret exists in each Modal environment, and the cliff_impacts grants in staging and production, which arrive with the Public API's deploy.
  • The US region-group calls read no database. v2_app.py changes only docstrings and comments.

Checks

  • Every test this PR adds or modifies, by node ID: in the contract library, the service library, the Stage 12 library, the executor and the Simulation Entrypoint. The database tests ran against a PostgreSQL 17 database built with the Public API's chain at a2b29deac (its cliff_impacts revision) and its privileges, connected as the Modal worker and service roles.

  • The repository's checks: Ruff format and lint, rules rendering, quality guards (file length included), lock checks, and Pyright for the executor, the Simulation Entrypoint and the three libraries.

  • Three review rounds. The first two found one bug each, both fixed test-first:

    • a null cliff_result_id in every context, which older workers refuse;
    • existing tests counting the sweep's kinds and the service's enum columns.

    The third found none.

🤖 Generated with Claude Code

anth-volk and others added 19 commits November 17, 2026 17:35
A cliff impact becomes each simulation run's own cliff result (C1 · 24's
answer 4b), so the report run's completion must neither set nor name
baseline_cliff_gap, baseline_cliff_share, reform_cliff_gap and
reform_cliff_share, and the Public API can then drop them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A report run's completion now sets only spm_provenance and data_version of
its own columns. Its update names none of the cliff columns, which stay
null, so the Public API can drop them (C1 · 41).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A cliff result belongs to one simulation run (C1 · 24's answer 4b). A
simulation worker's call that names one in its context claims it under
the call's Modal call ID, checks that its payload is that run's
simulation on the run's year and versions, calculates the simulation
with the cliff columns, and writes the weighted cliff gap and cliff share,
or its failure. A second start, or a restart after success, calculates
nothing. A US national calculation is split into the region groups a
plain run uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Public API's cliff_impacts table (C1 · 37's revision bc334f9877b7)
holds a simulation run's cliff calculations. Sim models it with the
service's tables: the simulation service inserts each pending row and
records its call; the Modal worker role claims it, and completes or fails
it, each in one conditional update. The Modal worker role no longer needs
to update report_runs' four cliff columns. A coordinator's call naming a
cliff result is refused before any write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
report_run_store.py was at 795 of its 800 lines. RunStatus,
ACTIVE_STATUSES, RunKind, RUN_TABLES, the row nouns, ActiveRun, RunWrite
and SimulationRunState move unchanged to run_states.py, and
report_run_store re-exports each, so every import of them keeps working.
No behaviour changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A simulation worker's call whose context names a cliff result
(Stage12InvocationContext.cliff_result_id) claims it in the Public API's
cliff_impacts under its Modal call ID, checks that the payload is the
simulation of the result's run on the run's year and versions, plans it
with .py's cliff columns and no others added, calculates it, and writes
.py's weighted cliff gap and cliff share, or the failure, only while its
call holds the row. A second start or a finished row calculates nothing.

A US national calculation runs as the 20 region groups a plain run uses,
whenever its run's own calculation would (should_segment_cliff_calculation):
each person's cliff columns depend only on their household, and the share
is taken once from the merged rows. A coordinator's call naming a cliff
result is refused.

Sim models cliff_impacts with the service's tables (C1 · 37's revision
bc334f9877b7), with each role's privileges; the Modal worker role no
longer needs to update report_runs' cliff columns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A cliff calculation left pending or running by a call that ended, or one
never started, would make every later request for the run's cliffs join
it. The sweep judges cliff results as it judges household jobs: after
the other kinds, by their recorded call, and by the age limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RunKind.CLIFF_RESULT names cliff_impacts, and the sweep judges it last,
as it judges household jobs: a cliff result pending with no call, or whose
recorded call ended, or past the age limit, is failed while it is as
listed, so the next request for the run's cliffs starts another.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Stage 12 foundation document, the two persistence libraries' READMEs,
the executor's README and the v2 app's workers say how a worker call naming
a cliff result calculates and records its run's cliff gap and share, how a
US national calculation is split, and that the sweep judges cliff results.
Adds the changelog fragment.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Stage12InvocationContext is strict, so a v2 application built before
cliff_result_id refuses a context carrying the field, even as null. The
simulation service serializes contexts with model_dump(mode="json"), so
until the new application is the default, every report run, simulation
run and budget-window year it starts would fail. Review round 1, bug 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The field is excluded when None (exclude_if), as the manifest's optional
local_area_reports is, so a context naming no cliff result serializes as
before and the v2 applications deployed before it accept the contexts the
simulation service sends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
As for a simulation run, a cliff calculation split into region groups
checks between polls whether its cliff result was taken from its call; a
read that fails is treated as not taken, and must leave a warning. Review
round 1 also asked for cover of a call without its Modal call ID, a
payload for another country's worker, the totals against .py's own
calculate_cliff_impact, and the cliff result's shared model in
RUN_TABLES; those tests pass already.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The check whether the cliff result was taken from the call logs a warning
when its read fails, as the simulation-run worker's check does, before
treating it as not taken.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The stuck-run sweep's application, store and summary, the executor
README's deployment step, the database-access skill and the v2 app's
secret comment now include simulation runs' cliff results. The Stage 12
foundation document says nothing starts a cliff calculation yet, how
C1 · 39 is to start one, and that the Public API's cliff_impacts revision
must be applied in an environment before this release deploys there.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three sweep tests counted three kinds of run per pass, and the enum test
seven enum columns among the service's tables; the sweep now also lists
cliff results, and cliff_impacts.status is an eighth. Review round 2,
bug 1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A budget-window year's context leaves out an unset cliff_result_id, and
the service role inserts a cliff_impacts row in its writes test. The
sweep's and run store's summaries, and the simulation workers' container
limit comment, now name cliff results. Review round 2's code-quality
findings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Twenty region-group results merged by the real segment code give the
same cliff gap and share as the unsplit rows. The foundation document's
segmentation section now names a simulation run's own cliff calculation
as split, and the Modal worker credentials' summary names cliffs.
Review round 3's code-quality findings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The fixtures list every privilege the simulation service needs, which now
include SELECT and INSERT on cliff_impacts and UPDATE of its call-start
columns. The bad grants still leave out the same two privileges.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Calculate each simulation run's cliff gap and share as its own cliff result

1 participant