Skip to content

validation: report checks that didn't run instead of passing them - #353

Draft
tnazarew wants to merge 2 commits into
ci/pr-coveragefrom
validation/no-false-greens
Draft

tnazarew wants to merge 2 commits into
ci/pr-coveragefrom
validation/no-false-greens

Conversation

@tnazarew

@tnazarew tnazarew commented Nov 5, 2026

Copy link
Copy Markdown
Collaborator

Stacked on #352 (base ci/pr-coverage); retarget to main after #352 merges.

Why

A green result has to mean the checks ran and passed. Today several things pass silently:

  • Hive results have never reached the report. Validation writes hive-dataproc--<ol>-report.json (inputs.hive_release doesn't exist), the upload looks for hive-dataproc-<cv>-<ol>-report.json and only warns (e.g. run 42009167037). report.json has no hive_dataproc entry, yet the job is green.
  • Hive's init action reads GCS_TRANSPORT_JAR_URL, CI sets GCS_TRANSPORT_JAR, so every Hive run used the 1.34.0 GCS transport.
  • A validator that raises returns None, the caller raises TypeError, and the whole component's validation is lost.
  • Scenarios are taken from the event directories, so a scenario that produced nothing isn't reported at all; zero events = zero syntax tests = green.
  • Semantic checks are skipped silently after a syntax failure or when no semantic test applies to the version.
  • One failed Spark cluster skips validation for every Spark pair.

What

Validator

Situation Now
scenario valid for the versions, no events FAILURE events: "scenario produced no events"
schema validator raised FAILURE of that event's syntax test ("validator error while checking …")
exception validating one scenario FAILURE validation for that scenario only
producer job not DONE (--run_status status.json) FAILURE producer job: "producer job ended in state …, job '…'" (replaces check-spark-jobs, so it's in the report)
syntax failed, semantics skipped SKIPPED semantics note
no semantic test applies to the version WARNING semantics

SKIPPED/WARNING don't make a scenario red, aren't "new failures" and aren't counted in the compatibility tables (report.py only looks at SUCCESS/FAILURE). --openlineage_release gates versions when the spec tag is main (snapshots).

Workflows

  • Spark: validate runs for every pair whose jobs ran (!cancelled()), even if another Spark release's cluster failed. A pair without status.json, or where the runner failed before any scenario, fails its validate job without a report. check-spark-jobs removed.
  • Hive: report path fixed; installs producer/hive_dataproc/runner/requirements.txt (was Spark's); a failed query fails its scenario instead of aborting the pair; writes status.json; init action reads GCS_TRANSPORT_JAR (old name kept as fallback).
  • dbt writes status.json; every report upload uses if-no-files-found: error.
  • Re-validation (ci: test shared code changes in PR checks #352) passes the stored status.json.

Hive expectations were stale and fail today: hardcoded hive-ol-cluster-m, _producer pinned to tree/1.34.0, SQL whitespace, and an input location (gs://…/warehouse/transactions) from an older query. test.sql creates the table without LOCATION, so it lives in the cluster's HDFS warehouse. They now match any Dataproc master and release. Dataset-level transformations are compared unordered (1.34.0 emits [SORT, GROUP_BY], 1.55.0 [GROUP_BY, SORT]). That needed a fix in compare_events.py: unordered_list never matched, because the module's own any() jinja helper shadows the builtin.

Verification

  • pytest scripts/tests → 43 passed. Written first against the old validator: 8 of the new validator tests failed for the expected reasons (TypeError crash, missing scenario, no note/warning, unknown options).
  • Stored Hive events (last release run, from GCS) with the new validator and expectations: 10/10 tests SUCCESS at OL 1.34.0 and 1.55.0 (main: 6/9 semantic tests failing at 1.55.0).
  • Stored Spark events of release run 42009167037 (6 pairs, all jobs DONE), old vs new validator: identical, 438 tests SUCCESS, no new failures vs report.json.
  • actionlint: no new findings (the pre-existing hive_release error is gone).
  • This PR's own check runs Spark, Hive, dbt, scenarios and dataplex for real; Hive has to be green there.

🤖 Generated with Claude Code

A green result has to mean the checks ran and passed.

Validator (scripts/validate_ol_events.py):
- Scenarios come from their definitions (valid for the validated
  versions) plus the event directories, so a scenario without events is a
  FAILURE ("scenario produced no events") instead of missing from the
  report.
- A schema validator that raises fails that event's syntax test; before,
  it returned None and the whole component's validation crashed with a
  TypeError.
- An exception while validating one scenario (bad JSON, unreadable config)
  fails only that scenario.
- Semantic checks skipped after a syntax failure get a SKIPPED note; a
  version pair without any applicable semantic test gets a WARNING. Neither
  makes a scenario red.
- --run_status <status.json>: a scenario whose producer job didn't end in
  DONE gets a "producer job" FAILURE. This replaces check-spark-jobs, so the
  signal is in the report.
- --openlineage_release: version gating when the spec tag isn't a version
  (snapshots validate against main).

Workflows:
- Spark validates every pair whose jobs ran, even when another Spark
  release's cluster failed; a pair that didn't run fails its validate job
  without a report.
- Hive: the report was written as hive-dataproc--<ol>-report.json but
  uploaded as hive-dataproc-<cv>-<ol>-report.json, so Hive results never
  reached the report. Fixed, plus: installs its own requirements, a failed
  query fails its scenario instead of aborting the pair, and the init
  action reads GCS_TRANSPORT_JAR (it read GCS_TRANSPORT_JAR_URL and always
  fell back to the 1.34.0 transport).
- Hive and dbt write status.json; report uploads fail when the report is
  missing; the re-validation passes stored status.json files.

Hive expectations were stale (hardcoded cluster name, _producer pinned to
1.34.0, SQL formatting, an input location from an older query) and fail
today; they now match any Dataproc master and release. Dataset-level
transformations are compared unordered (1.34.0 and 1.55.0 emit them in a
different order), which needed a fix in compare_events: unordered_list
never matched, because the module's own any() jinja helper shadows the
builtin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: tnazarew <tomasz.nazarewicz@getindata.com>
Released transports-gcs 1.40.0-1.55.0 break Hive on Dataproc: the
half-relocated Woodstox stops HiveServer2 from starting, and the bundled
Conscrypt native library aborts the Tez application master. Now that the
init action really installs the matching transport, the Hive 1.55.0
cluster can't start.

Point the clusters at transports-gcs-1.56.0-SNAPSHOT-conscrypt-fix.jar,
which has both problems fixed (verified: aggregate_job 10/10 on OpenLineage
1.34.0 and 1.55.0), and stop downloading and uploading the matching
transport. Revert once a released transport has the fix.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: tnazarew <tomasz.nazarewicz@getindata.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant