Repository navigation
Conversation
…House into iceberg_add_files
|
SET allow_insert_into_iceberg = 1; -- required for every add_files call Commands
ALTER TABLE my_iceberg_table ALTER TABLE my_iceberg_table ALTER TABLE my_iceberg_table ALTER TABLE my_iceberg_table |
|
TESTING |
CI triage for
|
| Check | Failing item | Classification |
|---|---|---|
| Integration tests (amd_asan_ubsan, targeted) | test_backward_compatibility/test_custom_s3_disk_remote_url_allow_hosts.py (2 tests) |
Pre-existing |
| Integration tests (amd_asan_ubsan, db disk, old analyzer, 7/8) | same 2 tests | Pre-existing |
| Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 2/2) | job timeout (12600 s) | Pre-existing / infra |
| Stateless tests (amd_binary, cas s3 storage, parallel) | 05055_format_schema_source_query_background_access |
Flaky |
| Regression release s3_azure_1 | Pre-existing | |
| Regression release settings | Pre-existing | |
| Regression release swarms | Pre-existing | |
| Regression release tiered_storage_cas | /tiered storage/with cas/simple replication and moves |
Environment (ENOSPC) |
| Regression release cas_lightweight_delete_4 | /lightweight delete/concurrent alter and delete/VersionedCollapsingMergeTree/concurrent clear update and delete |
Likely flaky (not PR code) |
1. test_custom_s3_disk_remote_url_allow_hosts.py: pre-existing
Both tests fail in fixture setup because the old-version node never comes up:
E Exception: Timed out while waiting for instance `node' with ip address 172.16.1.7 to start. Container status: running
The same two tests fail on unrelated PRs into antalya-26.8: #2881 (in both the targeted and 7/8 jobs) and #2885 (7/8). This is a backward-compatibility startup problem on the base branch and has nothing to do with Iceberg. Next step: none for this PR. It needs a separate fix or a skip on antalya-26.8.
2. Stateless ASan cas s3 storage 2/2: timeout, pre-existing
WARNING: Timeout exceeded [12600] for [27940]
WARNING: Job timed out: [Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 2/2)], timeout [12600], exit code [-15]
The shard got through 3514 of 6933 tests in 3.5 h and every test it ran was [ OK ]. The run did not hang; the shard is simply too slow for the budget. The same job ends in ERROR at about 12 690 s on #2881, #2885 and #2895 (in those PRs both shards 1/2 and 2/2 time out). Next step: none for this PR. The cas s3 storage ASan shards need more splits or a longer timeout on the base branch.
3. 05055_format_schema_source_query_background_access (amd_binary, cas s3): flaky
-background flush refused: 1
+background flush refused: 0
The CI's own diagnosis re-ran the test 14 times with the same randomized settings and got Failed: 0, Passed: 14. The test also fails on #2888, an unrelated PR. It is timing-dependent (Buffer background flush) and does not touch Iceberg. Next step: safe to re-run.
4. Regression s3_azure_1, settings, swarms: pre-existing
All three suites also fail on #2881 (an unrelated Iceberg compaction fix against the same base). Next step: none for this PR.
5. Regression tiered_storage_cas: environment
Code: 75. DB::Exception: ... Cannot write to file /jbod1/store/bc5/.../tmp_insert_all_6_6_0/columns_substreams.txt: , errno: 28, strerror: No space left on device
The small jbod1 test disk ran out of space during simple replication and moves. That is MergeTree moves, not Iceberg. The suite passed on #2881. Next step: safe to re-run.
6. Regression cas_lightweight_delete_4: likely flaky
One scenario out of 284 failed: concurrent alter and delete/VersionedCollapsingMergeTree/concurrent clear update and delete, with assert r.output == output at concurrent_alter_and_delete.py:341. It is a race-style MergeTree test and this PR touches no MergeTree or mutation code. The suite passed on #2881, so I can't call this one pre-existing, only flaky. Next step: re-run. If it fails again on a re-run, ping me and I'll look deeper.
Suggested fixes for the PR: none needed. No failure points at this PR's changes. If you want a cleaner board, re-run items 3, 5 and 6. Items 1, 2 and 4 will stay red until they are fixed on antalya-26.8.
Evidence: praktika result_pr.json for this SHA and for #2881, #2885, #2888 and #2895; shard job.logs; GitHub Actions logs for the regression jobs. I did not re-run anything locally.
mkmkme
left a comment
There was a problem hiding this comment.
🤖 [Posted by an LLM agent on behalf of @mkmkme]
Thanks for the feature. Several paths lead to wrong query results or let a user read data they couldn't reach before, so requesting changes. Details are inline. Points that don't map to one line:
- Files are unreadable by Spark/Trino. The registered files don't carry Iceberg field IDs, and no
schema.name-mapping.defaultis set. iceberg-java'sadd_filessets that mapping when it is missing. Without it, other engines map columns by position or read NULLs. The tests only read back through ClickHouse, even though they live intest_storage_iceberg_with_spark. - Tests needed: a Spark read of the registered files, with the file's column order different from the schema; a file without column stats in an identity-partitioned table, queried with
WHERE p = ...;DECIMAL,TIMESTAMP(MILLIS)and multibytetruncate[N]partitions, checked against a native insert of the same rows; a path outside the table location (should be rejected). - Nits: if a directory expands to zero files, an empty snapshot is committed;
std::stollon a malformed transform throws astd::exception, not a ClickHouse error; conflict retries end withBAD_ARGUMENTS; the changelog entry should mention the required setting and that only Parquet and non-bucket partitioning are supported.
| "Files for bucket-partitioned tables are not supported yet", field_name); | ||
|
|
||
| auto bounds_it = file.bounds_by_field_id.find(static_cast<Int32>(source_id)); | ||
| if (bounds_it == file.bounds_by_field_id.end() || !bounds_it->second.has_bounds) |
There was a problem hiding this comment.
A missing min/max doesn't mean the values are null. It also happens when the writer skipped stats, and for INT96, float, double and boolean columns (all Unsupported here). The file is then recorded with a NULL partition value, and WHERE p = x skips it, so the query returns wrong results. Please treat the value as NULL only when null_count == num_rows in every row group, and otherwise reject the file.
There was a problem hiding this comment.
Updated so derivePartitionField uses NULL only when row groups have null count equal to row count. Also added test.
| { | ||
| if (!bound.has_bounds) | ||
| continue; | ||
| if (bound.kind == SourceValueKind::Integer) |
There was a problem hiding this comment.
The byte width here comes only from the physical type. Iceberg stores decimal bounds as big-endian two's complement, so a DECIMAL stored as INT32 or INT64 gets corrupt bounds. TIMESTAMP(MILLIS) and TIMESTAMP(NANOS) are treated as microseconds, which breaks both the bounds and the day/hour/month partition values. FLBA decimals compared with std::min on raw bytes are wrong for negative values. Please check the Parquet logical type and either convert to Iceberg's encoding or drop the bounds (and reject the file if the column is a partition source).
There was a problem hiding this comment.
Added logic to reject
| { | ||
| if (bounds.kind == SourceValueKind::String) | ||
| { | ||
| String lo_trunc = bounds.min_str.substr(0, static_cast<size_t>(transform_argument)); |
There was a problem hiding this comment.
The Iceberg spec truncates strings to N code points, and our own icebergTruncate uses leftUTF8. substr cuts N bytes, so with multibyte text the partition differs from what a native insert would produce, and it can split a character. More generally, could we evaluate the existing partition expression from Utils.cpp on a two-row [min, max] block instead of re-implementing the transforms here? That would keep add_files consistent with native writes.
There was a problem hiding this comment.
Fixed. truncate[N] on strings now keeps N code points.
| result.registration_path = registration_path; | ||
|
|
||
| IcebergPathFromMetadata metadata_path = IcebergPathFromMetadata::deserialize(registration_path); | ||
| result.storage_path = resolver.resolve(metadata_path); |
There was a problem hiding this comment.
Nothing restricts the path to the table location. For an outside path, IcebergPathResolver::resolve falls back to stripping a prefix by its length, and then reads through the table's storage credentials. With only ALTER_EXECUTE, a user can register any Parquet object those credentials can reach, such as another table's data, and then read it with SELECT. For a different bucket, the manifest records s3://other/... while ClickHouse read some other key, so ClickHouse and Spark see different files. Please require paths under the table location, or add an explicit source-access check and match the resolved path exactly.
There was a problem hiding this comment.
Fixed. checkPathUnderTableLocation requires every path to start with the table location followed by /.
| analyzed_files.push_back(analyzeFile(path, resolver, object_storage, column_path_to_field_id, context, format_settings)); | ||
|
|
||
| /// Validate the file's columns against the table schema. | ||
| if (!allow_missing_columns) |
There was a problem hiding this comment.
Only "column present" is checked, not the type. A file with id String is accepted into an id Int32 table and only fails, or gives wrong bounds, at read time. allow_missing_columns = 1 also accepts files that lack a required field, which breaks the spec. Also, the message says the values "will read as the column default", but the test shows NULL.
There was a problem hiding this comment.
Updated checkParquetColumnType also checks the type.
| new_metadata_path, object_storage, persistent_components.metadata_cache, | ||
| context, log, new_compression, persistent_components.table_uuid); | ||
|
|
||
| if (metadata->getValue<Int64>(Iceberg::f_current_schema_id) != original_schema_id) |
There was a problem hiding this comment.
After a conflicting commit, we reload the metadata and re-check the schema and spec IDs, but we don't redo the duplicate check against the new snapshot. Two concurrent add_files calls on the same file can both commit, which double-counts the rows.
There was a problem hiding this comment.
Fixed. Added duplicate check in retry path.
| } | ||
| else if (command_name == "add_files") | ||
| { | ||
| checkTableRootIsQueriedPath("add_files"); |
There was a problem hiding this comment.
expire_snapshots and remove_orphan_files each have their own allow_* setting. add_files is only gated by allow_insert_into_iceberg. Given the risks above, could it get its own experimental setting, with a SettingsChangesHistory.cpp entry?
There was a problem hiding this comment.
Added new setting allow_experiemental_iceberg_add_files
…exactly (e.g. data for , ms/ns timestamps, , decimals), and reject unsupported Iceberg types, so footer statistics can be used as bounds as they are. - Use the partition only when the source column is absent or all its values are null; reject files without complete min/max statistics. Drop bounds and null counts that do not cover every row group. - Never delete the files of a snapshot that may be visible: with a non-transactional catalog, report a failed catalog update instead of cleaning up and retrying; with a transactional catalog, check whether the catalog applied the snapshot before cleaning up, and do not write the metadata file to storage. - Read the current metadata through the catalog (), and repeat the duplicate check after every retry. - Validate / arguments and handle identity partition values. - Add tests with pyarrow-written files for mismatched types, missing statistics and all-null partitions.
…ntal_iceberg_add_files
…House into iceberg_add_files
|
@mkmkme I would suggest that these 2 issues be addressed separately as more investigation is required on how the other iceberg-clients do this when the schema is changed. Issue 1:BackgroundParquet files without Iceberg field IDs, such as those registered by Problem
BackgroundParquet files without Iceberg field IDs, such as those registered by Issue 2:ProblemFor a file without field IDs, ClickHouse matches columns by their name in the file ( |
CI triage for 4501362 (
|
| Check | Classification | Caused by this PR? |
|---|---|---|
| Regression release settings | 1 new failure: allow_experimental_iceberg_add_files |
Yes, the suite's snapshot of setting defaults needs an entry (see below) |
| Regression release swarms | Pre-existing | No |
| Regression release s3_azure_1 | Pre-existing | No |
| RegressionTestsRelease / LightweightDelete / CASLightweightDelete / CASS3CacheLightweightDelete (3) | Hit the 5h job limit and were cancelled | No (infra/hang) |
| Integration tests (amd_asan_ubsan, targeted) | Flaky test_s3_cluster::test_graceful_shutdown |
No |
| Integration tests (amd_asan_ubsan, db disk, old analyzer, 3/8) | Same flaky test | No |
| Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 2/2) | Job timed out at 12600s with no test failures | No (infra) |
The PR's own tests pass: all 18/18 cases in test_storage_iceberg_with_spark/test_add_files.py PASSED in the targeted integration job.
1. Regression release settings: PR-related (snapshot only)
✘ [ Fail ] '/settings/default values/ai_function_max_concurrent_requests_per_thread'
✘ [ Fail ] '/settings/default values/ai_function_max_retries'
✘ [ Fail ] '/settings/default values/allow_experimental_iceberg_add_files' <-- new
✘ [ Fail ] '/settings/default values/json_extract_use_cast_string_to_date_time_mode'
✘ [ Fail ] '/settings/default values/output_format_arrow_max_record_batch_size_bytes'
The latest antalya-26.8 MasterCI run (44645958054, job Common (settings)) fails on the same four ai_function_* / json_extract_* / output_format_arrow_* settings. Only allow_experimental_iceberg_add_files is new. It's the setting this PR adds (DECLARE(Bool, allow_experimental_iceberg_add_files, false, ...)), and the regression suite's "default values" snapshot doesn't have an entry for it yet.
Fix: add allow_experimental_iceberg_add_files (default 0, Bool) to the settings default-values snapshot in the clickhouse-regression settings suite, the same way the other new 26.8 settings are being added. ClickHouse itself needs no change. Until then this check will stay red, but it's red on the base branch anyway.
2. Regression release swarms: pre-existing
✘ [ Fail ] '/swarms/feature/node failure/cpu overload'
✘ [ Fail ] '/swarms/feature/node failure/cpu overload all swarm nodes'
✘ [ Fail ] '/swarms/feature/feature/object storage cluster profile events with overloaded node'
The same node failure/cpu overload* scenarios fail on base MasterCI 44645958054 and on #2913. The extra profile events with overloaded node scenario covers the same overloaded-node area, and this PR doesn't touch swarm or cluster code. Not caused by this PR.
3. Regression release s3_azure_1: pre-existing
✘ [ Fail ] '/s3/azure/part 1/disk/remote host filter' (16m 57s)
✘ [ Fail ] '/s3/azure/part 1/disk/syntax' (5m 14s)
The same two scenarios fail on base MasterCI 44645958054 with near-identical timings, and on #2913. Not caused by this PR.
4. LightweightDelete (3) ×3 variants: timeouts
All three jobs (lightweight_delete_3, cas_lightweight_delete_3, cas_s3_cache_lightweight_delete_3) were cancelled with The job has exceeded the maximum execution time of 5h0m0s, so no report or logs were uploaded. On sibling PRs these suites usually take 1–1.5h, but #2917 also hit the 5h limit on lightweight_delete_3. This PR only touches Iceberg/DataLake code, which the lightweight-delete suites don't use. I'd call this a runner hang or slowdown. Safe to re-run. I can't confirm what hung, because the cancelled jobs left no logs.
5. Integration tests: test_s3_cluster/test.py::test_graceful_shutdown: flaky
test_s3_cluster/test.py:2171 - in test_graceful_shutdown
assert errors == 0
E assert 1 == 0 (targeted) / assert 2 == 0 (db disk, old analyzer, 3/8)
This test kills a swarm node mid-query and relies on fixed sleeps. It fails in the same way on #2928 (a CI-only change, same db disk, old analyzer, 3/8 shard) and on #2913 (amd_msan, 10/10). It has nothing to do with Iceberg add_files. Not caused by this PR.
6. Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 2/2): job timeout
[6373 / 7003] 05064_alias_marker_in_lambda_over_distributed: [ OK ] 7.40 sec.
WARNING: Job timed out: [Stateless tests (amd_asan_ubsan, cas s3 storage, parallel, 2/2)], timeout [12600], exit code [-15]
6373 of 7003 tests finished before the job was killed, and the log has no [ FAIL ] lines. The CAS stateless lanes time out regularly on this branch: the MSan CAS lanes ERROR on base MasterCI and on #2913 and #2917, and #2926 is splitting these lanes because of it. Infra, not caused by this PR.
Next steps
- Add
allow_experimental_iceberg_add_filesto the settings regression snapshot (item 1). This is the only failure the PR is responsible for. - Optionally re-run the LightweightDelete (3) jobs. Everything else matches known base-branch or flaky failures.
Note: the build jobs on this head reused cached binaries from 5796fd6, so the tests above ran against that build. That's fine: the only commit since then (4501362 Fix failing tess) changes nothing but test_add_files.py.
closes: #2215 ClickHouse#128228
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Added ALTER TABLE ... EXECUTE add_files(...) for Iceberg tables to register existing Parquet files into a new snapshot without
rewriting them.
CI/CD Options
Exclude tests:
Regression jobs to run: