Repository navigation
Conversation
Add an internal reader for the Mumbling bitmap format, which v4 manifest deletion vectors use (manifest_info.dv, tracking.deleted_positions and tracking.replaced_positions). The spec is still a draft, so the header is internal and not installed. MumblingBitmapView wraps the serialized bytes without copying them and supports Contains() and Cardinality(). Make() validates everything that memory safety depends on, so lookups never read out of bounds. Test: - New unit tests in util_test (mumbling_bitmap_internal_test.cc) cover the spec examples, Java-generated fixtures and malformed input. - The full test suite passes on both the SSE2 and the portable lookup path, and the new tests pass under ASan and UBSan. - New microbenchmarks in src/iceberg/benchmark (mumbling_bitmap_benchmark, built with -DICEBERG_BUILD_BENCHMARKS=ON).
xndai
force-pushed
the
xndai/iceberg-cpp/B1
branch
from
November 14, 2026 19:36
ce1a7d1 to
b11c29a
Compare
Use KeepRunning() loops and an explicit main() in the benchmark (as in benchmark_smoke.cc), std::cmp_less for signed/unsigned comparisons, and std::array instead of a C array in the test.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1125. Part of #1107.
Adds an internal reader for the Mumbling bitmap format (a draft spec, apache/iceberg#16518), which v4 manifest deletion vectors use (
manifest_info.dv,tracking.deleted_positionsandtracking.replaced_positions). Because the spec is a draft, the header isutil/mumbling_bitmap_internal.hand isn't installed.MumblingBitmapView::Make()wraps the serialized bytes without copying them. In scan planning those bytes belong to the root manifest entry, which outlives the planning of its leaf manifest.Make()checks the header, the PFOR-encoded descriptor array (Appendix A), the descriptor types, and the container sizes against the buffer, soContains()never reads out of bounds. Like the Java reader, it doesn't validate container contents. Sparse-container lookups use SSE2 on x86-64 and a portable SWAR search elsewhere; no compiler flags are added.Tests cover the spec examples, malformed input (including the vectors from Java's
TestMumblingBitmap), and the Java-generated fixtures from apache/iceberg-go#1750.mumbling_bitmap_benchmark(built with-DICEBERG_BUILD_BENCHMARKS=ON) measures decoding and lookups.Benchmark against the Java reader on the same input bytes (Release; gcc-14 with Google Benchmark, JDK 17 with JMH; Xeon Platinum 8175M):
Make, 100,000 random positionsMake, all 2,097,152 positionsContains, 4,096 probes, 100,000 random positionsContains, 4,096 probes, all positionsTest:
the spec examples, Java-generated fixtures and malformed input.
path, and the new tests pass under ASan and UBSan.
(mumbling_bitmap_benchmark, built with -DICEBERG_BUILD_BENCHMARKS=ON).