Repository navigation
Stable 26.3 Backport of #128217 - Evaluate the PARTITION BY expression of file and object storage writes once per INSERT, not once per block - #2908
Conversation
…tateful-function-spans-chunks Evaluate the `PARTITION BY` expression of file and object storage writes once per INSERT, not once per block Conflicts committed as-is; the next commit resolves them: - src/Storages/IPartitionStrategy.cpp (both modified) - src/Storages/IPartitionStrategy.h (both modified)
Adapted the PR to stable-26.3's non-const strategy API, which has no `getCachedOrBuildActions`, and kept a one-argument `computePartitionKey` for the DeltaLake sink.
|
Port-AL: cherry-pick conflicts resolved by AI Conflicted files:
Commits:
|
CI triage for 82ec992Verdict: 4 failing checks. None are caused by this PR. 1 is infra (crash during install, before any test ran), 2 are pre-existing (Grype CVE scans, same result on #2899), and 1 is expected for backports (DCO). The infra failure did one more thing: praktika dropped about 30 downstream jobs (integration, stress, fuzzers, sequential stateless, SQLLogic). Coverage for this backport is incomplete until that job is re-run.
1. Stateless tests (amd_asan, distributed plan, parallel, 3/4): infraThe only failing "test" is Evidence that the PR didn't cause it:
Next step: re-run this job (or the failed jobs of run 44344552760). The re-run should also bring back the dropped integration, stress, fuzzer and sequential suites, which this storage-sink change should get before merge. I could not confirm from the report that the new 2. Grype scans (keeper, server-alpine): pre-existing
3. DCO: expected for backportsCommit Suggested fixes for PR-related failuresNone needed. Re-run the ASan 3/4 job so the dropped suites run. |
CI triage for
|
| Check | Classification | Next step |
|---|---|---|
| Integration tests (amd_binary, 5/5) | Infra: the runner died | Re-run |
| SQLLogic test | Timeout, already seen on other stable-26.3 PRs |
Re-run, or ignore |
| DCO | Sign-off missing on 2 commits | Maintainer override / remediation commit (see below) |
1. Integration tests (amd_binary, 5/5): infra
- praktika:
error — Job failed to produce Result due to a script error or CI runner issue. No test results were uploaded. - Check-run annotation (job 135705930223):
The self-hosted runner lost communication with the server. Verify the machine is running and has a healthy network connection.
- The job died in the
Runstep after about 13 minutes, before any test result was recorded. Shards 1 to 4 were not executed in this run. Shard 5/5 also failed on Stable 26.3 Backport of #131963 - Clear--jwtand the other credential options from the command line ofclickhouse-clientandclickhouse-keeper-client#2931 (an unrelatedstable-26.3backport) on 2026-11-20. - Next step: re-run the job. We have no test names, so this verdict comes from the runner-loss annotation only.
2. SQLLogic test: timeout, already seen on other stable-26.3 PRs
- From
sqllogic_test/job.log:statements-testfinished (16:33 to 17:25). Thencomplete-testwas still running when the job was killed:No statement mismatch or error was reported before the kill.[2026-11-19 19:32:03] WARNING: Timeout exceeded [10800] for [43115] [2026-11-19 19:32:06] WARNING: Job timed out: [SQLLogic test], timeout [10800], exit code [137] - Other
stable-26.3PRs hit the same 3h timeout: Stable 26.3 Backport of #131001 - Reuse the GCM cipher context between rows in encrypt/decrypt #2899 (failed after 3h01m, 2026-11-18) and Stable 26.3 Backport of #131154 - Reject out-of-rangeDecimalscale when reading aField#2850 (failed after 3h01m, 2026-11-13). When the job does pass it is close to the limit: Stable 26.3: Bump to 26.3.64 #2872 took 2h20m and Stable 26.3 Backport of #131963 - Clear--jwtand the other credential options from the command line ofclickhouse-clientandclickhouse-keeper-client#2931 took 2h34m. This job is borderline on this branch whatever the PR changes. This PR only touches thePARTITION BYwrite path for file and object storage (PartitionedSink,IPartitionStrategy), which SQLLogic does not use. - Next step: re-run if a green check is needed. Otherwise ignore it. A longer-term fix for the branch would be to raise the SQLLogic timeout or shard the job (separate from this PR).
3. DCO: sign-off missing
dcd0269"Merge pull request Evaluate thePARTITION BYexpression of file and object storage writes once per INSERT, not once per block ClickHouse/ClickHouse#128217 …" (upstream merge commit by Alexey Milovidov):Sign-off not found82ec992"Resolve conflicts in backport of Evaluate thePARTITION BYexpression of file and object storage writes once per INSERT, not once per block ClickHouse/ClickHouse#128217" (blau-ai):Sign-off not found0eda6e1merge fromstable-26.3: skipped (merge commit)- The upstream merge commit comes straight from ClickHouse/ClickHouse, so it cannot be fixed by amending without rewriting history. The DCO bot's own suggestion is a rebase plus force-push, which I can't do and which I don't recommend for a backport branch.
- Next step: a maintainer can mark DCO as passed / override it, as for other backports, or the author can add an individual remediation commit (see the dco2 README). Not code-related.
Bottom line: nothing to fix in the code. Re-run the integration shard (and SQLLogic if you want it green). DCO needs a maintainer decision.
|
Correction to my triage above, under 1. Integration tests (amd_binary, 5/5): the sentence "Shards 1 to 4 were not executed in this run" is wrong. In this run, amd_binary 1/5 to 4/5 and amd_asan (db disk, old analyzer) 1/8 to 8/8 all passed. Only 5/5 errored, because its runner was lost. The verdict stays the same: infra, safe to re-run. |
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Fixed
PARTITION BYoffile,urland object storage writes (table functions and table engines) evaluating stateful and non-deterministic functions anew for every block since 26.1:PARTITION BY intDiv(rowNumberInAllBlocks(), N)restarted its counter on every block and wrote rows into the wrong files, andnow64()produced one file per block instead of one perINSERT. Closes ClickHouse#128211. (ClickHouse#128217 by @groeneai)Documentation entry for user-facing changes
...
CI/CD Options
Exclude tests:
Regression jobs to run: