Skip to content

Answer 500 when a household job's create may have been committed, and require the SQLAlchemy ORM (ratcheted) - #1090

Draft
anth-volk wants to merge 14 commits into
mainfrom
v2plan/c1-33-household-create-outcome-unknown
Draft

anth-volk wants to merge 14 commits into
mainfrom
v2plan/c1-33-household-create-outcome-unknown

Conversation

@anth-volk

@anth-volk anth-volk commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1089

Summary

Two changes: a 500 for a household job whose commit may have been applied, and a new rule, ratcheted, that database code uses the SQLAlchemy ORM (below).

When the database fails after a household job's row was inserted, PostgreSQL may already have applied the commit, so the job may exist. POST /household-calculations now answers that case with the JSON 500 internal_error instead of 503 database_unavailable, the way the economy and household report routes already do (#1052).

  • Persistence: HouseholdJobStore.create_job tells a failed commit from a failed insert. An error while the insert runs propagates as before (nothing written; a refused reference is still HouseholdJobReferenceError). An error once the insert has run is the commit's: it raises the existing EconomyRecordOutcomeUnknownError, caused by the commit's error, as household_report_store and economy_run_store do.
    • economy_run_store imports household_job_store (for UnknownCatalogVersionError), so create_job imports the error inside the method, with the reason on the PLC0415 suppression. economy_run_store.py is unchanged.
  • Service: HouseholdCalculationService._create turns that error into the new HouseholdCalculationOutcomeUnknownError (the commit error's type), before any Modal call is started. Every other SQLAlchemyError still answers 503 database_unavailable and logs household_job_not_created.
  • Route: answers {"code": "internal_error", "detail": "The service may have created the household job, but could not confirm it was written."} with status 500 and no Retry-After, and logs the fixed event household_job_outcome_unknown at error level with error_type in policyengine_attributes, as economy_record_outcome_unknown does. The record names no job ID: the observability runtime drops every attribute whose key contains household, and the observability guide reserves job_id for a Modal invocation. The runtime adds the submission's observability_id.
  • Docs: the 500 and 503 OpenAPI descriptions, the error-code comments, the module docstrings and both READMEs now say that a submission's 503 means no job will run and that its 500 may follow a write. A fixed changelog fragment.

If the job was written, it stays pending with no Modal call ID, and the stuck-run sweep fails it.

Rule: database code uses the SQLAlchemy ORM (ratcheted)

This PR also adds sim.persistence.orm-only. Code that reads or writes database rows declares tables only as SQLAlchemy ORM mapped classes, and runs every statement through an ORM Session against them, never through Core Table objects or Connections. Today every database module uses Core, so the rule starts ratcheted. #1097 tracks the conversion.

  • Check: projects/policyengine-simulation-entry/tests/orm_access_support.py reads every file the database boundary rule scans. It looks for Core rather than for the ORM, so any statement on mapped classes passes. It resolves names within each module, through that module's own imports. A Core table is a name bound to Table(...) or table(...). It is also a name imported from a module that exports one (directly, through a package __init__ or through *), a module attribute such as tables.household_jobs, a name annotated as SQLAlchemy's Table, or a name or attribute assigned from one. It reports:

    • Core table declarations: Table, Column, table, column, the clause classes, and MetaData outside an orm_base.py;
    • every reference to a Core table, which covers its statements, methods, column collections and joins;
    • the routes from a mapped class to its table: __table__, local_table, persist_selectable, a mapper's or MetaData's tables, and any .table read in a module that imports sqlalchemy;
    • Connection imports anywhere, and Engine or create_engine outside a library's engine.py;
    • .connect(), .connection(), .get_bind() and .raw_connection() calls, and .begin() on an engine;
    • raw SQL, except a server default, an index condition or a check constraint;
    • sqlmodel imports.

    test_every_table_is_mapped_by_a_class also imports both persistence libraries and reports every table no mapped class maps. The docs list what the check doesn't follow: importlib, tables passed around without a Table annotation, tables held in a container, attribute reads in other modules, and Session.bind.

  • Baseline: rules/baselines/orm-only.txt lists today's 880 findings as file: check: source, without line numbers. tests/test_orm_only.py fails on a finding the baseline doesn't list, and on an entry that no longer matches one, so the list only shrinks. The registry entry has [rule.downgrade] naming Move sim's database code to the SQLAlchemy ORM #1097, which the rules guard requires for a new rule that starts ratcheted.

  • Proof: rules/fixtures/sim.persistence.orm-only/ holds a bad fixture, the household job records as Core code (shortened from today's), and a good one, the same records on a mapped class and a Session. test_rule_proof_orm_only asserts each fixture's exact findings, and that the runtime check reports the bad fixture's table only. tests/test_orm_access_support.py has unit tests for each check, for the ORM forms it accepts, and for ordinary code it must leave alone.

  • Related rule: sim.stage12.persistence-typed-dml-only said "typed SQLAlchemy Core DML only", which the new rule contradicts. It now says "typed SQLAlchemy statements only". Its checks are unchanged.

  • Removed: policyengine_fastapi.database, which nothing imports, and which was the only sqlmodel import. The sqlmodel dependency of policyengine-fastapi stays until Move sim's database code to the SQLAlchemy ORM #1097's last step, which relocks the projects that install it.

  • Docs: an ORM section in docs/engineering/skills/database-access.md, which covers:

    • what the ORM means here: models, sessions, statements, naming, and how the deployment schema comparison reads the schema without a finding;
    • what the check reports and what it doesn't.

    Also re-rendered docs/generated/rules.md and AGENTS.md, and an added changelog fragment.

  • For other open PRs: once this merges, a PR that adds Core code fails test_database_code_uses_only_the_orm. A PR that changes existing Core code updates that code's baseline entries; the test's message lists them.

How the Public API reads the 500

No Public API change is needed. policyengine_api/libs/simulation_household_calculations.py on master (_accepted_job) reads sim's 500 as SimulationServiceInternalError with the "may have created the household job" message, which asks the caller to submit again and says this creates a second job if the first request went through. It reads sim's own coded 503 as SimulationServiceUnavailableError, "no job of the submission will run". So with this change, a lost commit is no longer reported as nothing created.

Tests

Written first and seen failing (2d32be72 for the household job fix, acf03f1e for the rule), then passing, each run by node ID:

  • libs/policyengine-simulation-persistence/tests/test_household_job_store.py

    • test_a_job_whose_commit_fails_may_have_been_created
    • test_a_job_whose_insert_fails_was_not_created
    • test_a_job_naming_missing_rows_is_refused_before_its_commit, against PostgreSQL 17 with the Public API's v2 schema from master, in a real top-level transaction: the keys refuse the insert statement, so a refusal is never taken for a lost commit.
  • projects/policyengine-simulation-entry/tests/test_household_calculations.py

    • test_a_job_whose_commit_is_unconfirmed_may_exist_and_starts_no_call
    • test_a_job_whose_insert_fails_is_logged_by_the_service
  • projects/policyengine-simulation-entry/tests/test_household_routes.py

    • test_a_job_whose_commit_is_unconfirmed_answers_a_500_and_is_logged
  • projects/policyengine-simulation-entry/tests/test_orm_only.py

    • test_database_code_uses_only_the_orm
    • test_every_table_is_mapped_by_a_class
    • test_rule_proof_orm_only
  • projects/policyengine-simulation-entry/tests/test_orm_access_support.py: each of its unit tests, 64 cases.

  • projects/policyengine-simulation-entry/tests/test_database_access_boundaries.py::test_every_source_directory_is_scanned, which the new rule also lists, unchanged.

The refused-reference (400 missing_reference), failed-insert route (503) and normal-create cases are covered by existing tests, which this PR doesn't change. Ruff, python -m scripts.rules guard --base-ref origin/main, the quality guards, the lock checks and Pyright pass.

Review

One read-only review round found no bugs. Two of its quality findings are fixed (5d145807): the store tests take their failing-transaction fakes from a new support module, tests/transaction_fakes.py, instead of adding another copy; and the insert-failure service test checks only the event the service emits. The third copy of the insert-then-wrap create logic is left for the ORM conversion (#1097), which rewrites the stores' create methods.

The ORM rule took eight further read-only review rounds (rounds 2 to 9).

  • Rounds 2 to 8: each found bugs, which were fixed before the next round. The main ones:
    • the first version flagged valid ORM statements;
    • one repository-wide set of table names mixed up unrelated modules;
    • several routes to Core went unreported: table methods, aliases, module attributes, mappers, class attributes, tuple assignments and a column's .table;
    • following class attributes across modules matched unrelated names.
  • Round 9: found no bugs. Its wording and test findings are fixed in 696df102.

No Dockerfile, workflow, image or Terraform changes, and no new environment variable or secret.

🤖 Generated with Claude Code

anth-volk and others added 5 commits November 17, 2026 11:50
When the commit fails after the job's row was inserted, PostgreSQL may
already have applied it. The household job store is to raise
EconomyRecordOutcomeUnknownError then, the service
HouseholdCalculationOutcomeUnknownError with the job's ID and the commit
error's type, and the route a JSON 500 internal_error, logged as
household_job_outcome_unknown. Guard tests: a failed insert statement and a
refused reference keep their own errors, and a failed insert still answers
database_unavailable and logs household_job_not_created.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the database fails after the household job's row was inserted,
PostgreSQL may already have applied the commit, so the job may exist. The
household job store now tells a failed commit from a failed insert and
raises the existing EconomyRecordOutcomeUnknownError, as the economy and
household report creates do. The service starts no call and raises
HouseholdCalculationOutcomeUnknownError, and POST /household-calculations
answers the JSON 500 internal_error, which the Public API reads as a job
that may exist, and logs household_job_outcome_unknown with the commit
error's type. 503 database_unavailable now means no job was written.

economy_run_store imports household_job_store, so create_job imports the
error inside the method rather than at module level.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e emits

Review round 1's quality findings. The household job store's tests take
their commit-failing and statement-failing transactions from a new support
module instead of adding another copy. The insert-failure service test now
checks only that the service logs household_job_not_created, the event the
unconfirmed-commit test checks is absent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The proof fixtures for the new rule sim.persistence.orm-only: the bad one is
the Core household job records the persistence library has today, shortened;
the good one is the same records on a mapped class and a Session. The rule's
tests fail until its check, orm_access_support.py, exists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk and others added 9 commits November 17, 2026 19:59
Add sim.persistence.orm-only: code that reads or writes database rows
declares tables only as ORM mapped classes and runs every statement through
an ORM Session against them. orm_access_support.py checks the source (Core
tables, column collections, engines and connections outside a library's
engine.py, statements not on a mapped class, raw SQL and sqlmodel), and
test_every_table_is_mapped_by_a_class imports both persistence libraries and
reports each table no mapped class maps.

The rule starts ratcheted: rules/baselines/orm-only.txt lists the 818
findings in today's Core code, the test fails on any other finding and on an
entry that no longer matches one, and issue #1097 tracks the conversion.
sim.stage12.persistence-typed-dml-only no longer says "Core", which the new
rule contradicts; its checks are unchanged. The unused
policyengine_fastapi.database module, the only sqlmodel import, is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 2's fixes. The statement check accepted only a mapped class or
a SQLAlchemy call as a statement's first argument, so it rejected ORM
statements on labels, exists(), variables, select(*columns) and subquery
columns; and Core reached through table(), a MetaData's tables, a mapper's
local_table or session.get_bind() passed. The check now flags what reaches a
Core table instead: a statement or select_from/join with a Core table as an
argument (a name bound to Table() or table(), or imported under another
name), .c or .columns of one, __table__, local_table, persist_selectable,
sorted_tables and a MetaData's tables, table() and column(), and get_bind().
Mapped classes no longer need recognising in source, which removes the
disagreement with the runtime check over DeclarativeBaseNoMeta; the runtime
check accepts both bases.

Unit tests cover each check, and the baseline is regenerated: 811 entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 3's fixes. One repository-wide set of table names made ordinary
names such as regions and pyarrow's Table count as Core tables in unrelated
modules, while a table's own methods, Core aliases, module attributes and
session.query() went unreported, and engine.begin() opened a Connection with
no finding. The check now resolves Core tables within each module, through
its own imports (relative ones and module attributes included), SQLAlchemy
Table annotations and names assigned from a table, and reports every
reference to one, which covers statements, methods, column collections and
joins in one check. begin() on an engine is reported, and so is importing
Connection even in a library's engine.py.

Baseline entries now name the table referenced instead of whole statements;
regenerated, 876 entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 4's fixes. The check now resolves a dotted module path
(import a.b without as, a package's re-exports, star imports), counts names
and attributes assigned from a module's table (jobs = tables.household_jobs,
self.jobs = household_jobs), and reports tables and selectable of a mapper
however it is reached (inspect(), class_mapper(), object_mapper(), .mapper
or a name bound to one), a mapped attribute's table, and the TableClause,
ColumnClause and TextClause constructors. Two files with one module name are
refused instead of one replacing the other. begin() counts as opening a
Connection only on a name that is an engine, not on any name containing
"engine". Unit tests cover each route, and the runtime check's
DeclarativeBaseNoMeta path; the baseline is unchanged at 876 entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 5's fixes. A class attribute declared as a table, such as a
dataclass field table: Table, now counts as one wherever it is read, in its
module and in modules that import from it, so shared_rows' index.table,
economy_rerun_records' key.table and the Stage 12 report run store's
result_rows.table are reported. A tuple assignment pairs each name with its
value. A mapper held in a name assigned __mapper__ or .mapper is followed.
Annotating self.x as a Table no longer makes the bare name x one. The rules
that go by a method or attribute name alone (.table, .connect(), .cursor(),
.begin() on an engine) apply only in modules that import sqlalchemy.

Unit tests cover each route; the baseline is regenerated: 880 entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 6's fixes. Matching a class attribute declared as a table in
every module that imported from its module went by the attribute's name
alone, so it reported unrelated reads (economy_rerun_records' key.table is a
column) and, because one module's exports fed another's, could miss an
export or never settle. A class attribute now counts only within the module
that declares it; another module's read isn't followed, and the reference
that stored the table there is reported where it was made. Once a library's
declarations are gone, every route to a Core table starts at a reported
reference such as __table__, so the end state needs no cross-module tracking.

The docs list the eight rules that apply only in modules importing
sqlalchemy, the cross-module limit, and how the schema comparison inspects
the database through inspect(engine) in engine.py. Baseline: 877 entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 7's fix. A mapped attribute's .table was reported only on a
chain rooted at a capitalised name, as in HouseholdJob.id.table, so a column
held in a name or parameter (key.table, job_id.table) reached its Core table
unreported, contradicting the docs' claim that every route to a Core table
starts at a reported reference. Every .table read in a module that imports
sqlalchemy is now reported, except a .table that is called, such as the
sa.table(...) constructor. This brings back the Stage 12 report run store's
two result_rows.table reads and economy_rerun_records' key.table, which is a
column's table.

Nested tuple assignments are paired too. The docs say how the schema
comparison reads the models' columns and keys without a finding, and that
instance attributes, like class attributes, aren't followed into other
modules. Unit tests cover .table through names and parameters, nested
tuples and an import cycle. Baseline: 880 entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 8's fix. Every .table read is reported, so a converted field
that still holds its model under the name table, such as ResultRows.table,
would stay a finding. The docs and the rule test's failure message now say to
name such an attribute after what it holds, such as model. The docs also say
which reads of another module's attributes are reported, that a called
.table isn't a read, and that the end state reports the ordinary routes to a
Core table, with the listed limits as the exceptions. Unit tests cover a
called and an assigned .table, and both modules of the import cycle.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 9 found no bugs; these are its quality findings. The naming
guidance and the failure message say that .table reads are reported in
modules that import sqlalchemy; the limits say tables and engines without a
Table or Engine annotation, and no longer claim mapper routes the check
doesn't follow. A CheckConstraint's condition may be SQL text, as a server
default and an index condition may. The import-cycle test now needs a second
pass over the modules to pass.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk anth-volk changed the title Answer 500, not 503, when a household job's create may have been committed Answer 500 when a household job's create may have been committed, and require the SQLAlchemy ORM (ratcheted) Nov 17, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Answer 500, not 503, when a household job's create may have been committed

1 participant