Skip to content

feat: allow bounded trailing gaps during production promotion - #6159

Draft
gjesse wants to merge 3 commits into
SQLMesh:mainfrom
gjesse:jh/5825-bounded-trailing-gaps
Draft

gjesse wants to merge 3 commits into
SQLMesh:mainfrom
gjesse:jh/5825-bounded-trailing-gaps

Conversation

@gjesse

@gjesse gjesse commented Oct 21, 2026

Copy link
Copy Markdown
Collaborator

Related to #5825.

A production plan can finish its planned backfill and then fail the no-gap check because a scheduled run advanced the incumbent snapshot while the replacement was being built. This adds an opt-in policy that allows promotion and finalization when the replacement is missing only a bounded, contiguous trailing suffix created after the plan was built. The suffix stays missing in interval state so the normal scheduler can repair it.

Strict no-gap behavior remains the default (0). The GitHub bot can opt in with max_trailing_gap_intervals_on_deploy_to_prod=1; Python callers can use Context.plan_builder(..., no_gaps=True, max_trailing_gap_intervals=1). The underlying plan/promotion policy is independent of GitHub. This PR does not add a CLI flag or notebook magic option.

Safety and scope

  • Capture the incumbent snapshot identity and frontier with the original plan bounds, and preserve them through plan serialization and apply.
  • Require complete candidate coverage through that frontier. Historical/internal gaps, changed schedules, ambiguous boundaries, over-bound suffixes, and conflicting plans still fail.
  • Count scheduler advances rather than lookback-expanded backfill work, including partial intervals and schedules whose cron differs from their storage interval unit.
  • Reject overlapping recorded coverage in transitive downstream candidates, and emit structured acceptance/rejection warnings.
  • Use the existing promotion/finalization path without a full-plan retry, scheduler pause, or new interval invalidation mechanism.

This is a bounded freshness tradeoff rather than the model-level locking proposed in the issue. Subsequent scheduler selections must include upstream dependencies. The bound applies to drift observed by the promotion check; an already-running scheduler is not fenced from advancing the old snapshot afterward. This change does not add durable repair tracking.

Validation

  • 202 focused promotion, policy, repair integration, state-sync, and GitHub CI/CD tests passed.
  • 376 broader regressions passed across state sync, context, plans, evaluation stages, and GitHub CI/CD (overlaps the focused run).
  • After correcting typed access to structured log fields, all 104 state-sync tests passed again.
  • make style passed: Ruff, Ruff formatting, mypy, and migration validation.

The implementation has also run in our production fork. Two production deployment attempts successfully used the allowance. For one, operations metadata and retained interval state corroborate subsequent repair on the exact promoted snapshot: the covering scheduler batch completed about 4m18s after finalization. This verifies one model's repair; it does not establish all downstream row values or a durable repair-time guarantee.

Signed-off-by: Jesse Hodges <hodges.jesse@gmail.com>
…ecks

Signed-off-by: Jesse Hodges <hodges.jesse@gmail.com>
Signed-off-by: Jesse Hodges <hodges.jesse@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant