Skip to content

Take data_source on /v2/reports, answer each report's dataset_independent_key, and find reports by it - #4310

Merged
anth-volk merged 18 commits into
masterfrom
v2plan/c1-30-data-source-and-keys
Nov 17, 2026
Merged

anth-volk merged 18 commits into
masterfrom
v2plan/c1-30-data-source-and-keys

Conversation

@anth-volk

@anth-volk anth-volk commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #4309

Summary

Builds on #4303 (merged 2026-11-16, e96e52ffb; it adds reports.dataset_independent_key and ix_reports_dataset_independent_key), and on #4307 (merged 2026-11-17, c5b8aab32): #4307 stops the Public API sending dataset_name to the simulation service and makes POST /v2/reports and POST /v2/simulations refuse it, so this PR adds no dataset_name change of its own. #4313 (per-user report runs, merged 2026-11-17, 05332d347) merged first, so this PR merged master and resolved the overlap: the search by dataset_independent_key takes the same optional user credential as the list and reads each report the same way (the user's saved label and description, the run of their newest request, and a 401 for a browser credential that names a signed-in user).

  • data_source on POST /v2/reports. An economy report takes data_source: "local_area_dataset", left out unless given (as spm): an explicit null or another value is refused with 422, and budget windows and household reports refuse the field as unknown. It is sent to the simulation service only when given. The Public API checks only the value; the simulation service checks the country (uk), the region (uk and the four country/* nations) and whether the release gives local areas a dataset of their own. Its 422s answer 422 with its code and reasons, and its 400 local_areas_unsupported and local_area_dataset_mismatch (its own state) answer 503, through the existing client mapping. The value is defined once (ReportDataSource, LOCAL_AREA_DATASET).
  • dataset_independent_key on every report read. Every report item of every kind (detail, list, search, and the create and re-run answers) answers the stored reports.dataset_independent_key, null for a report stored before the keys. The Public API never computes a key.
  • The search: GET /v2/reports?country_id=…&dataset_independent_key=…. The list route takes either report_id (repeated, at most 25, as before) or one dataset_independent_key (64 lowercase hexadecimal digits), never both or neither (422). It answers the country's reports with that key as summary items, newest created first (ties to the larger ID), at most the 25 created last, through ix_reports_dataset_independent_key. No credential, as every report read; the runtime role's existing SELECT on reports.
  • A local-area report's read. A UK-wide or nation economy report whose dataset is the default dataset of both a constituency region and a local-authority region of its catalog version is a local-area report, as the simulation service decides which dataset local areas use; nothing stores the request's data_source. Its read says data_source: "local_area_dataset" (null for every other economy report), and its result adds constituency_withheld_codes and local_authority_withheld_codes from withheld_area_codes (Record the UK area codes disclosure control withholds from a report run #4305), sorted, [] when none were withheld; an area level whose areas were all withheld reads []. Every other result leaves the two lists out, as the simulation service's SingleYearMacroOutput does (Calculate UK-wide and nation reports on the local-area dataset, with their areas and withheld codes policyengine-sim-api#1068). The withheld codes are read only for a local-area report.

What must merge and deploy first:

Migration: no route group moves; /v2/reports stays on FastAPI. User-visible contract changes: the new request field, the new query parameter (and report_id becoming optional when the key is given), and the new read fields. The generated migration contract documents are unchanged by the export.

Inline suppression added: # noqa: PLR0913 on store_report in tests/integration/stored_report_rows.py (the connection and catalog, then each column a test varies): it gains two keyword arguments for the key and the creation time, and grouping its existing arguments would change every caller.

Testing

Each change was built test first (the failing tests are their own commits). Run by node ID only:

  • unit: tests/unit/v2/test_report_dataset_independent_keys.py (SQLite: every kind of read answers the key; the search's country, order, ties, limit and empty answer), tests/unit/v2/test_local_area_report_reads.py (SQLite: which reports read as local-area reports), tests/unit/v2/test_economy_result_rebuild.py (withheld codes rebuilt sorted; an empty level; other results leave the lists out), tests/unit/v2/test_report_read_routes.py (the key, the search route, its refused queries, OpenAPI), tests/unit/v2/test_report_create_routes.py and tests/unit/libs/test_simulation_reports.py (data_source passed on and refused; the simulation service's refusals, as stand-ins shaped as its C1 · 29a answers);
  • integration, PostgreSQL 17 as the runtime role (tests/integration/test_v2_report_reads.py): the search; a local-area report's areas and withheld codes; an empty level; UK-wide and nation reports on the national dataset;
  • lint-imports, python -m scripts.rules validate, python scripts/run_quality_guards.py, ruff, ruff format, mypy, and python scripts/export_migration_contracts.py (no change). No revision or grant change.

Two read-only review rounds: round 1 found no bugs (documentation, one shared value, test setup and coverage fixed); round 2, on those fixes, found no bugs (test and documentation fixes only).

🤖 Generated with Claude Code

anth-volk and others added 11 commits November 14, 2026 02:39
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y it (failing)

Every report read answers reports.dataset_independent_key, and
GET /v2/reports takes dataset_independent_key in place of report_id to
find the country's reports that share it, newest first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every report item, of every kind, answers reports.dataset_independent_key
as the simulation service stored it, null for a report stored before the
keys. GET /v2/reports takes dataset_independent_key in place of
report_id and lists the country's reports that share it, newest created
first, at most the 25 created last, through
ix_reports_dataset_independent_key. The query takes one of the two.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e (failing)

POST /v2/reports takes data_source: "local_area_dataset" on an economy
report and sends it on only when given; null, another value and the
field on a budget window or household report are refused before the
simulation service is asked. The simulation service checks the region,
the country and the release: its 422 answers 422, and its 400 for its
own local-area dataset state 503.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
EconomyReportFields takes data_source: "local_area_dataset", left out
unless given, as spm is; an explicit null or another value is refused
with 422, and budget windows and household reports refuse the field as
unknown. The report request sends it only when given. The simulation
service checks the region, the country and the release, and its
refusals map as before: a 422 answers 422, and a 400 about its own
local-area dataset 503.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A UK-wide or nation report whose dataset is the one the catalog version
gives both the constituency and local-authority regions is a local-area
report: its read says data_source local_area_dataset, and its result
lists each level's withheld area codes, sorted, with a level whose areas
were all withheld read as empty. Every other result leaves the two lists
out, as the simulation service's does. The result fixture stores
withheld_area_codes as PolicyEngine/policyengine-sim-api#1068's writer
does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A UK-wide or nation report whose dataset is the default dataset of both
the constituency and the local-authority regions of its catalog version
is a local-area report, as the simulation service decides which dataset
local areas use. Its read says data_source local_area_dataset, its
result lists each level's withheld area codes from withheld_area_codes,
sorted, and a level whose areas were all withheld reads as empty. Every
other report reads data_source null, and its result leaves the two lists
out, as the simulation service's SingleYearMacroOutput does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The two report items' docstrings keep both sides: the null label and the
dataset-independent key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The request's ReportDataSource and its LOCAL_AREA_DATASET value now name
the read's data_source too. The list result's docstring gives the
search's order, and the read's data_source says it comes from the
report's region and dataset. The stand-in refusals match the simulation
service's answers, the two SQLite read tests share their setup
(tests.unit.v2.sqlite_report_rows, which drops one suppression), and
PostgreSQL now reads a UK-wide report on the national dataset and a
local-area report with an empty area level.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The simulation service's two data_source refusals now answer the UK
requests that draw them: a UK-wide report on a release without a
local-area dataset, and a constituency's own report. The value's comment
says a local area's own report never reads it, and the economy item's
docstring reads in order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 13, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.43750% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 93.67%. Comparing base (c5b8aab) to head (ff3a294).
⚠️ Report is 20 commits behind head on master.

Files with missing lines Patch % Lines
...licyengine_api/fastapi_routes/v2/reports/routes.py 88.88% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##           master    #4310   +/-   ##
=======================================
  Coverage   93.67%   93.67%           
=======================================
  Files         397      397           
  Lines       22831    22885   +54     
  Branches     3589     3599   +10     
=======================================
+ Hits        21387    21438   +51     
- Misses        840      845    +5     
+ Partials      604      602    -2     
Flag Coverage Δ
integration 57.69% <82.81%> (+0.06%) ⬆️
unit 90.95% <98.43%> (+0.09%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Base automatically changed from v2plan/c1-25-shared-report-keys to master November 16, 2026 17:05
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk and others added 2 commits November 17, 2026 15:20
Resolves the overlap the two tasks agreed C1 · 30 resolves: the search by
dataset_independent_key takes the list route's optional user credential
and reads each report through the same requester view, so a search shows
the user's saved label and newest recorded run as a list does, and refuses
a browser credential naming a signed-in user with a 401. The route's list
or search choice moves into a helper to keep the router builder within
ruff's complexity limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#4307's commits were already here; its own merge of #4313 resolved the
report-request docstring without data_source. This branch's wording keeps
master's and adds data_source, so it stays. The merged tree is identical
to the previous head's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk
anth-volk marked this pull request as ready for review November 17, 2026 13:12
@anth-volk
anth-volk merged commit 7e188ab into master Nov 17, 2026
24 checks passed
@anth-volk
anth-volk deleted the v2plan/c1-30-data-source-and-keys branch November 17, 2026 13:12

This branch was previously deployed

1 inactive deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Take data_source on /v2/reports, answer each report's dataset_independent_key, and find reports by it

1 participant