Repository navigation
Conversation
Route the default single-worker deferred reader through FileIO range coalescing and stream reuse. Preserve payload order, duplicate descriptors, nulls, table-scoped I/O, and configured merge bounds. Add request-range and stream-lifecycle regression tests and adapt the existing table-scoped FileIO test double to the shared read path. Co-Authored-By: Codex <noreply@anthropic.com> AI-Model: gpt-6 Co-Authored-By: Codex <noreply@ai-tool.com> AI-Contributed/Feature: 8/8 AI-Contributed/UT: 81/81
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
The Python deferred BLOB reader bypasses FileIO range coalescing when
blob_parallelismis 1, including the default. It reads each descriptor separately, even when descriptors point to adjacent or overlapping ranges in the same file.Use the existing
read_blobs_concurrentpath for one worker as well. This lets serial reads merge ranges and reuse a stream while preserving result order, duplicate descriptors, nulls, table-scoped FileIO, and configured merge limits. For the regression fixture, the default path changes four individual payload reads into one merged range using one stream.This change is limited to the Python deferred scalar BLOB reader; the Rust reader and the configured parallelism are unchanged.
Tests
PYPAIMON_TEST_EXECUTION=python python -m pytest -q pypaimon/tests/deferred_blob_resolve_test.py pypaimon/tests/blob_test.py: 205 passed (Python 3.12.14, PyArrow 19.0.1).