Repository navigation
Conversation
CI and the new local script call the same runner scripts: - runner/cluster.json: image and Hive properties per Hive release (were inline in the workflow) - runner/create_cluster.sh: creates the cluster with dev/dataproc/install_jars.sh, uploaded per cluster instead of to a shared path that concurrent runs overwrote; keeps the fixed transports-gcs snapshot as the default - runner/run_scenarios.sh: one Hive job per scenario, continues past failures, records the job's state and id in status.json - run_hive_dataproc_tests.sh: one command for a local run (jar from Maven, cluster deleted at the end, validation) run-hive-query now exits non-zero when the job doesn't end in DONE, so a failed query is never recorded as DONE. Removes runner/get_hive_openlineage_jars.sh and the unused properties in the run step. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: tnazarew <tomasz.nazarewicz@getindata.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #358 (and with it #356, #353, #352); retarget after those merge.
The Hive Dataproc workflow and a local run now call the same scripts, so a scenario can be checked on your own cluster before it's committed.
What changes
producer/hive_dataproc/runner/cluster.json: the Dataproc image and Hive properties per Hive release. These were inline in the workflow'screate-commandandcase.runner/create_cluster.sh: creates the cluster with the shareddev/dataproc/install_jars.shinit action. Each cluster uploads its own copy togs://<bucket>/scripts/<cluster>/. Before, every run uploaded to one sharedscripts/path, so concurrent runs could overwrite each other's init action. The fixedtransports-gcssnapshot from validation: report checks that didn't run instead of passing them #353 is now the script's default, so CI and local runs use the same transport. Released 1.40.0–1.55.0 still break Hive.runner/run_scenarios.sh: runs one Hive job per scenario and keeps going when one fails. It records each job's final state and job id instatus.json, for exampleERRORwith the job id, where it used to recordFAILEDwith no job id.dev/dataproc/dataproc_workflow.py run-hive-query: exits non-zero unless the job ends inDONE, and prints the final state. A job that ends inCANCELLED, or one that is still running when the wait times out, can no longer be recorded asDONE.propertiesare gone.run_hive_dataproc_tests.sh(was empty): one command for a local run. It gets the jar from Maven (or a local build via--hive-jar), creates a cluster named after you and deletes it at the end, then runs the scenarios and validates. The README has the step-by-step playbook.runner/get_hive_openlineage_jars.sh.Tests
scripts/tests/test_hive_runner.py:create_cluster.shandrun_scenarios.shwith stubpython/gcloud: arguments, per-cluster init action, unknown release, andstatus.jsonafter a done job, a failed job, a failed submit and a missing query.run-hive-query's exit code for DONE, ERROR, CANCELLED and RUNNING. The last three fail on the old code.test_run_scripts.py:--help, unknown options and missing arguments for the three new scripts.pytest scripts/tests: 105 passed. shellcheck is clean, and actionlint reports nothing on the changed lines.🤖 Generated with Claude Code