Skip to content

hive: run the scenarios locally the way CI does - #361

Draft
tnazarew wants to merge 1 commit into
local-runs/dbtfrom
local-runs/hive
Draft

tnazarew wants to merge 1 commit into
local-runs/dbtfrom
local-runs/hive

Conversation

@tnazarew

Copy link
Copy Markdown
Collaborator

Stacked on #358 (and with it #356, #353, #352); retarget after those merge.

The Hive Dataproc workflow and a local run now call the same scripts, so a scenario can be checked on your own cluster before it's committed.

What changes

  • producer/hive_dataproc/runner/cluster.json: the Dataproc image and Hive properties per Hive release. These were inline in the workflow's create-command and case.
  • runner/create_cluster.sh: creates the cluster with the shared dev/dataproc/install_jars.sh init action. Each cluster uploads its own copy to gs://<bucket>/scripts/<cluster>/. Before, every run uploaded to one shared scripts/ path, so concurrent runs could overwrite each other's init action. The fixed transports-gcs snapshot from validation: report checks that didn't run instead of passing them #353 is now the script's default, so CI and local runs use the same transport. Released 1.40.0–1.55.0 still break Hive.
  • runner/run_scenarios.sh: runs one Hive job per scenario and keeps going when one fails. It records each job's final state and job id in status.json, for example ERROR with the job id, where it used to record FAILED with no job id.
  • dev/dataproc/dataproc_workflow.py run-hive-query: exits non-zero unless the job ends in DONE, and prints the final state. A job that ends in CANCELLED, or one that is still running when the wait times out, can no longer be recorded as DONE.
  • Workflow: the create and run steps are now one script call each. The "Upload initialization actions" step and the run step's unused properties are gone.
  • run_hive_dataproc_tests.sh (was empty): one command for a local run. It gets the jar from Maven (or a local build via --hive-jar), creates a cluster named after you and deletes it at the end, then runs the scenarios and validates. The README has the step-by-step playbook.
  • Removed runner/get_hive_openlineage_jars.sh.

Tests

  • scripts/tests/test_hive_runner.py:
    • create_cluster.sh and run_scenarios.sh with stub python/gcloud: arguments, per-cluster init action, unknown release, and status.json after a done job, a failed job, a failed submit and a missing query.
    • run-hive-query's exit code for DONE, ERROR, CANCELLED and RUNNING. The last three fail on the old code.
  • test_run_scripts.py: --help, unknown options and missing arguments for the three new scripts.
  • pytest scripts/tests: 105 passed. shellcheck is clean, and actionlint reports nothing on the changed lines.

🤖 Generated with Claude Code

CI and the new local script call the same runner scripts:
- runner/cluster.json: image and Hive properties per Hive release (were inline in the workflow)
- runner/create_cluster.sh: creates the cluster with dev/dataproc/install_jars.sh, uploaded per cluster instead of to a
  shared path that concurrent runs overwrote; keeps the fixed transports-gcs snapshot as the default
- runner/run_scenarios.sh: one Hive job per scenario, continues past failures, records the job's state and id in
  status.json
- run_hive_dataproc_tests.sh: one command for a local run (jar from Maven, cluster deleted at the end, validation)

run-hive-query now exits non-zero when the job doesn't end in DONE, so a failed query is never recorded as DONE.
Removes runner/get_hive_openlineage_jars.sh and the unused properties in the run step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: tnazarew <tomasz.nazarewicz@getindata.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant