Skip to content

dbt: run the scenarios locally the way CI does - #358

Draft
tnazarew wants to merge 1 commit into
dev/shared-helpersfrom
local-runs/dbt
Draft

tnazarew wants to merge 1 commit into
dev/shared-helpersfrom
local-runs/dbt

Conversation

@tnazarew

@tnazarew tnazarew commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #356 (dev/ helpers). Part of the robustness work: contributors should be able to test a scenario on their machine before committing, with the same steps and scripts as CI.

Why

producer/dbt/run_dbt_tests.sh didn't do what CI does. It needed a local OpenLineage checkout, validated every run as dbt 1.8.0 (hardcoded), ran every scenario regardless of version, and installed packages into whatever environment was active. The CI run loop was inline YAML, so nothing could share it.

What

  • producer/dbt/runner/run_scenarios.sh --events-dir DIR --scenarios 's1;s2' runs each scenario's test/run.sh and records DONE/FAILED in status.json with dev/lib/status.sh. A failed scenario doesn't stop the others and doesn't fail the script; validation fails it with a "producer job" test (validation: report checks that didn't run instead of passing them #353). CI's run step is now this one call. It no longer has continue-on-error, so only an infrastructure error (for example, a missing jq) fails the step.
  • producer/dbt/run_dbt_tests.sh --component-version V --openlineage-version OL [--scenarios] [--events-dir] [--report-dir] is plain and linear, with the CI steps in order:
    1. a cached virtualenv with the same packages as CI
    2. the valid scenarios (scripts/get_valid_test_scenarios.sh)
    3. run_scenarios.sh
    4. dev/validation/validate_locally.sh, which writes dbt-V-OL-report.json (the CI artifact name) and exits 1 if a scenario failed
  • producer/dbt/README.md "Run locally, as CI does" covers prerequisites, picking versions, the one command, the same steps by hand (one command per CI step, labelled with its name), where the output lands, how to read a failure, and how to stop the Postgres container.
  • Template: run_template_tests.sh has the same shape with TODOs, and README.md is a skeleton of the sections. producer/README.md lists the dbt command.
  • .gitignore: the default output (/*-events-*/, /*-report.json) when run from the repo root.

Verification

  • pytest scripts/tests: 87 passed. The new scripts' --help, unknown option and missing arguments are covered by test_run_scripts.py.
  • shellcheck: clean on the changed scripts. actionlint: no findings on the changed lines.
  • Local run of run_dbt_tests.sh --component-version 1.8.0 --openlineage-version 1.55.0 (Python 3.12, like CI) with Docker stubbed to fail:
    • the venv has dbt-core 1.8.0, dbt-postgres 1.9.1 and openlineage-dbt 1.55.0
    • csv_to_postgres was picked and recorded as FAILED
    • the report has "scenario produced no events" plus "producer job ended in state FAILED"
    • the script exited 1
  • The hand-step validation on this PR's real dbt 1.55.0 CI events fails only the 3 assertion tests that dbt: expect the current dataQualityAssertions shape #357 fixes.
  • Not yet: a full local run against Postgres. The Docker engine on my machine doesn't respond, so this waits for a working Docker. Also not yet: a branch dispatch of the release check with only dbt.

🤖 Generated with Claude Code

@tnazarew
tnazarew force-pushed the local-runs/dbt branch 2 times, most recently from 82c2500 to 4b1d51b Compare November 13, 2026 19:13
- producer/dbt/runner/run_scenarios.sh runs the scenarios and records how each ended in status.json (dev/lib/status.sh).
  The CI run step now calls it instead of inline YAML; a failed scenario is recorded, not a failed step.
- producer/dbt/run_dbt_tests.sh is rewritten as a plain local run with the same steps as CI: a cached virtualenv with
  dbt-core==V, dbt-postgres and openlineage-dbt==OL, the scenarios valid for the versions, run_scenarios.sh, and
  dev/validation/validate_locally.sh. It no longer needs an OpenLineage checkout or hardcodes dbt 1.8.0.
- README: "Run locally, as CI does", with the one command, the same steps by hand (one per CI step) and how to read the
  output. The template producer gets the same script shape and a README skeleton; producer/README.md lists the command.
- .gitignore: the local run's default output in the repo root.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: tnazarew <tomasz.nazarewicz@getindata.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant