[JingsongLi]

Purpose

Make Native execution the default with the released pypaimon-rust 0.5.0 wheel. Install pypaimon-rust>=0.5.0 automatically on Python 3.10+ and enable native planning, reading, writing and committing for supported operations. Preserve explicit Python selection and existing fallback; Python 3.6–3.9 continues to use Python execution.

Respect explicit planner, reader and writer opt-outs independently. Indexed reads preserve the selected Python reader, and Python writes keep row-ID routing in Python. Perform compatibility checks within Native preparation so disabled backends and fork-safety errors retain their dispatch behavior.

Preserve Java/Python contracts where the released runtime needs Python execution:

• Ordinary fixed-bucket append writers use Python because Native 0.5.0 emits zero sequence bounds, which changes Java compaction order. Primary-key and bucket-unaware Native writers remain available; Native publication remains independent.
• Data Evolution commits containing retained row-ID files use Python publication to validate written field identities against each publication attempt's schema. Native writes and update preparation remain available. Regression tests cover dropped/re-added fields, stale handles, omitted conflict baselines and rejected-commit cleanup.
• Java ORC shared-shredding MAP files use Python reads, including retained files after format/layout changes. Qualify effective and historical file schemas before opening a stream. Parquet shared-shredding and ordinary ORC routes retain Native execution.
• Configured shredded VARIANT writes and updates use Python to retain nested Parquet field IDs required by Java. This includes unmatched upsert/MERGE inserts outside scalar update projections. Plain VARIANT, scalar direct updates and delete-only operations retain Native support.

Run modern Python CI against the released wheel and maintain a dedicated Python Execution job. Keep full-suite Native coverage gates out of short interoperability sessions and bootstrap pip before dependency-group installation. Use explicit Python baselines and normalize unordered primary-key scan results in tests.

Use spawn for multi-worker Torch DataLoaders to avoid inheriting Polars worker locks. Preserve writer-failure injection across both backends and update installation, execution and multiprocessing documentation.

Tests

• Native CI regression configuration with real Rust 0.5.0: 444 passed, including planner/read/write/commit counters and all seven update coverage gates. Broader writer/update/MERGE tests: 1,931 passed; VARIANT/read-type/sidecar tests: 255 passed, 1 skipped.
• Python Execution regressions: 295 passed, 1 skipped.
• Real Java 8 append-compaction-order and plain/shredded VARIANT read tests passed. Actual Java ORC files passed 16 public reads across format/layout overrides and full/selected MAP projections. The affected Parquet/ORC/Avro primary-key round trips passed.
• Independent review: no remaining actionable findings in the repaired dispatch and compatibility boundaries. Twenty-one interoperability regressions cover sequence restoration, physical MAP/VARIANT layout and schema-probe/fork failures. An independent 29-case matrix verifies VARIANT aliases, batch/stream, insert bypasses and retained Native scalar/delete operations.
• Complete Torch suite: 79 passed with the process default forced to fork. LeRobot/preference tests passed in both execution lanes.
• actionlint, Flake8 and git diff --check passed.

The complete GitHub Actions matrix passed for commit 691cf96a86, including Native Python 3.10–3.13, Python Execution, Java/Python interoperability and all Java jobs.

[Akash3121]

commit_table_messages inspects message.new_files here before calling owner._prepare_native_commit(non_empty_messages) (and even when commit.native.enabled is false).

This is the direct cause of the 5 failing tests in both Python / Native (3.10–3.13) and Python / Python Execution:

• native_write_fork_safety_test.py::test_native_preparation_propagates_fork_errors_without_python_mutations[direct-commit]
• native_write_fork_safety_test.py::test_native_preparation_propagates_fork_errors_without_python_mutations[cause-commit]
• native_write_fork_safety_test.py::test_native_preparation_propagates_fork_errors_without_python_mutations[context-commit]
• native_write_fork_safety_test.py::test_ordinary_native_preparation_failures_keep_the_safe_python_fallback[ordinary-error-commit]
• native_write_fork_safety_test.py::test_ordinary_native_preparation_failures_keep_the_safe_python_fallback[missing-runtime-commit]

In native_write_fork_safety_test.py (lines 60 - 68), operation == 'commit' creates message = Mock() with message.is_empty.return_value = False on a data-evolution table and patches native_messages_supported. Iterating for file in message.new_files here raises TypeError: 'Mock' object is not iterable before _prepare_native_commit is ever reached.

In addition, retained row-ID column identity validation only applies to partial-column updates (file.write_cols is not None), not every file with first_row_id is not None. Moving this check into native_messages_supported (or checking file.first_row_id is not None and file.write_cols is not None inside native_messages_supported / prepare_native_commit) keeps commit message validation in one place and fixes all 5 CI jobs:

Suggested change

	# the schema used by each publication attempt. Python performs that check
	prepared = owner._prepare_native_commit(non_empty_messages)

[Akash3121]

Flipping both SCAN_NATIVE_PLAN_ENABLED and READ_NATIVE_ENABLED defaults to True here breaks single-option opt-out (scan.native-plan.enabled=false) in native_plan_requested() (paimon-python/pypaimon/read/read_dispatch.py:408-415):

def native_plan_requested(scan) -> bool:
    """Resolve Native planning preference without loading a disabled backend."""
    if scan.table.options.native_plan_enabled():
        return True
    if not scan.table.options.native_read_enabled():
        return False
    from pypaimon.execution.native.runtime import native_runtime_available
    return native_runtime_available()

Previously, when both options defaulted to False, falling through to scan.table.options.native_read_enabled() allowed read.native.enabled=true to also opt into native split planning. Now that READ_NATIVE_ENABLED defaults to True, if a user explicitly sets scan.native-plan.enabled=false (without also setting read.native.enabled=false), scan.table.options.native_plan_enabled() returns False, but scan.table.options.native_read_enabled() still returns True from its new default—so native_plan_requested(scan) still returns True and ignores scan.native-plan.enabled=false (which is also why the tests in this PR had to add 'read.native.enabled': 'false' alongside 'scan.native-plan.enabled': 'false').

Similarly, configure_indexed_read(table) in paimon-python/pypaimon/read/read_dispatch.py:250-259 still checks if not any(options.contains(option) for option in (CoreOptions.READ_NATIVE_ENABLED, CoreOptions.SCAN_NATIVE_PLAN_ENABLED)): and copies {CoreOptions.READ_NATIVE_ENABLED.key(): 'true'} into table.options.options, which overrides the PYPAIMON_TEST_EXECUTION=python / @pytest.mark.python_read test-lane default in conftest.py:246-252 whenever pypaimon_rust is installed.

If SCAN_NATIVE_PLAN_ENABLED is explicitly configured in scan.table.options, native_plan_requested(scan) should respect it directly:

def native_plan_requested(scan) -> bool:
    """Resolve Native planning preference without loading a disabled backend."""
    if scan.table.options.options.contains(CoreOptions.SCAN_NATIVE_PLAN_ENABLED):
        return scan.table.options.native_plan_enabled()
    if not (scan.table.options.native_plan_enabled() or scan.table.options.native_read_enabled()):
        return False
    from pypaimon.execution.native.runtime import native_runtime_available
    return native_runtime_available()

[Akash3121]

With write.native.enabled, read.native.enabled, and scan.native-plan.enabled all defaulting to True now, setting write.native.enabled=false alone does not select the Python execution path in create_row_id_plan_execution() (paimon-python/pypaimon/write/backend_dispatch.py:351-360):

def create_row_id_plan_execution(table, plan):
    """Select pure routing once; execution errors never replay another backend."""
    if (table.options.native_write_enabled() or table.options.native_read_enabled()
            or table.options.native_plan_enabled()):
        from pypaimon.execution.native.write.row_id_plan import create_native_row_id_plan, NativeRowIdPlan
        native = create_native_row_id_plan(table, plan)
        if native is not None:
            return NativeRowIdPlan(table, plan, native=native)
    from pypaimon.execution.python.write.row_id_plan import PythonRowIdPlan
    return PythonRowIdPlan(plan)

Previously, when all three options defaulted to False, native_write_enabled() or native_read_enabled() or native_plan_enabled() worked as an opt-in check. Now that all three default to True, setting write.native.enabled=false leaves native_read_enabled() and native_plan_enabled() as True, so create_row_id_plan_execution() still routes row-ID updates (update_by_row_id) to NativeRowIdPlan unless the user disables all three options.

Should create_row_id_plan_execution() check if table.options.native_write_enabled(): (or respect an explicit write.native.enabled=false) so that setting write.native.enabled=false selects PythonRowIdPlan as documented here?

[leaves12138]

Reviewed the Native-default switch and the explicit Python opt-outs, missing-runtime fallback, and compatibility guards for bucketed append sequence ranges, shredded VARIANT field IDs, ORC shared-shredding MAP reads, and retained-row-ID update publication.

Validated against the released pypaimon-rust==0.5.0 runtime on Python 3.10:

• Native regression selection: 1,077 passed, exercising native planning, reads, writes, commits, and all seven update categories.
• Python-execution selection: 334 passed; 743 Native-only cases skipped as expected.
• CI command-boundary test: passed, including both execution modes and success/failure propagation.

All current GitHub check runs are successful. I did not rerun the full Java interoperability or Torch suites locally. No blocking issues found in the reviewed head.