Skip to content

[python] Coalesce deferred BLOB reads with one worker - #11155

Open
YannByron wants to merge 1 commit into
apache:masterfrom
YannByron:codex/python-single-worker-blob-coalescing
Open

YannByron wants to merge 1 commit into
apache:masterfrom
YannByron:codex/python-single-worker-blob-coalescing

Conversation

@YannByron

Copy link
Copy Markdown
Contributor

Purpose

The Python deferred BLOB reader bypasses FileIO range coalescing when blob_parallelism is 1, including the default. It reads each descriptor separately, even when descriptors point to adjacent or overlapping ranges in the same file.

Use the existing read_blobs_concurrent path for one worker as well. This lets serial reads merge ranges and reuse a stream while preserving result order, duplicate descriptors, nulls, table-scoped FileIO, and configured merge limits. For the regression fixture, the default path changes four individual payload reads into one merged range using one stream.

This change is limited to the Python deferred scalar BLOB reader; the Rust reader and the configured parallelism are unchanged.

Tests

  • PYPAIMON_TEST_EXECUTION=python python -m pytest -q pypaimon/tests/deferred_blob_resolve_test.py pypaimon/tests/blob_test.py: 205 passed (Python 3.12.14, PyArrow 19.0.1).
  • The three new parameterized cases fail against the unmodified upstream implementation and pass with the change. They cover default merging, a zero-gap limit, a maximum merged-span limit, duplicate/out-of-order descriptors, nulls, and closing the reused stream.
  • Flake8 with the repository configuration passed for all three changed files.

Route the default single-worker deferred reader through FileIO range
coalescing and stream reuse. Preserve payload order, duplicate descriptors,
nulls, table-scoped I/O, and configured merge bounds.

Add request-range and stream-lifecycle regression tests and adapt the
existing table-scoped FileIO test double to the shared read path.

Co-Authored-By: Codex <noreply@anthropic.com>
AI-Model: gpt-6
Co-Authored-By: Codex <noreply@ai-tool.com>
AI-Contributed/Feature: 8/8
AI-Contributed/UT: 81/81
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant