[JustinBinber]

Purpose

Spark Data Evolution MERGE and read planning can exhaust Driver heap with many small column files. Metadata accumulates across scan planning, split serialization, task-result decoding and commit preparation; removing one eager collection alone can move the failure later in the same SQL.

Add an opt-in table mode, data-evolution.metadata-memory-optimization.mode=off|lightweight|streaming (default off). Lightweight reduces eager decoding, copies and overlapping object lifetimes. Streaming additionally sorts/replays metadata, incrementally plans complete RowID dependency groups, and uses compact summaries/external identity reconciliation in the actual commit path. Partial-write tasks produce messages on demand. One atomic Snapshot and existing compatibility contracts are preserved. data-evolution.metadata-memory-optimization.max-disk-size limits each metadata store; it is not a global process disk budget.

Scope: the planning-to-commit chain required by the reproduced Data Evolution MERGE failure, plus the separately validated controlled wide-table read-planning case. This does not establish repair of the separate production three-million-file SELECT/LIMIT incident or constant memory. Task arrays, retained result bytes, largest messages/groups, RowID summaries and callback/deletion-vector/diagnostic fallback paths remain boundaries.

Capacity and exact correctness

Fixed clean comparison base: b7cfb397de334d56b8c3eaeb60a2cfa9c9cab627. Candidate: 3d64be92a161a12130ede4f2719fb4df4a1b8844. Runtime production/tests/build inputs are unchanged from frozen f17caf4cc27eaa87927fe0fb28385462aecb373d; the later commit only regenerates configuration HTML. Normal existing artifacts are reused with applicable-input and runtime-origin checks, not claimed rebuilt for that documentation commit.

With 524288 real two-row base files, Driver 1 GiB and two 512 MiB Executors, clean upstream fails with Driver heap OOM during scan planning and publishes no Snapshot. Same-input/same-resource streaming completes the full MERGE in 740.262 s and publishes only Snapshot 513. No speedup is calculated against the failure. A separate unmodified-upstream oracle (4 GiB Driver/two 1 GiB Executors, disclosed separately) passes five exact bidirectional checks: full rows, unchanged source, old key/RowID pairs, complete RowID domain and inserted RowID domain (1153433 result rows). Seed/source and earlier Snapshot bytes remain unchanged.

Successful-case performance

One 4-core/16-GiB Ubuntu host, Spark 3.5.9/Scala 2.12.18, JDK17 SQL processes; this is not a multiple-worker-node benchmark. At each scale all 16 measured MERGEs and five-check oracles pass. Four balanced Williams blocks, fixed inputs/SQL/heaps/split settings, fresh private copy and JVM per arm, no concurrent build/archive/analysis during timed blocks. Oracles use a separate 2-GiB Driver and are outside timing. Cache state is not identical/cold; balancing reduces but does not eliminate carryover. All samples are retained.

Base files	Mode	SQL median seconds	vs upstream median
32	upstream	7.950692	+0.00%
32	off	7.737670	-2.68%
32	lightweight	7.800494	-1.89%
32	streaming	8.083136	+1.67%
65536	upstream	150.318557	+0.00%
65536	off	150.177861	-0.09%
65536	lightweight	150.161253	-0.10%
65536	streaming	107.755112	-28.32%

These four samples per mode do not establish p95, significance or general throughput gains. Streaming's first paired 32-file block is +11.42%; its small-case sampled Driver heap is higher. Larger successful-case gains coexist with increased local writes and commit work. The linked results include every sample, CPU/deserialization/GC, heap/RSS, commit duration, shuffle/Spark spill and common-live process I/O. Spark spill excludes connector-local sort/replay; process I/O is sampled and not exact SQL/connector-only I/O, and raw parent/child counters must not be summed. Exact isolated encoding/planning CPU is unavailable.

Separate controlled SELECT/LIMIT acceptance

Frozen wide input: 32768 real two-row base files, 259 business columns, one 128-column partial update, Snapshot 33 and 72089 result rows. The reader projects all business columns plus _ROW_ID, and verifies actual values, NULLs, domains and counts. At the same 512-MiB Driver/two 512-MiB Executors, clean fixed-b7 fails both LIMIT 20 and full read with Driver Java heap OOM in Manifest decoding/decompression propagated through read planning. Streaming completes LIMIT 20 (9.573720134 s) and full read (82.889682472 s), without Snapshot/input changes. This is an independent controlled case, not a diagnosis of the production exit137 incident. Five exact bidirectional checks run separately with a 2-GiB Driver through each actual tested reader. The initial streaming controller stopped at an unstarted exact-oracle disk preflight; its successful reads were retained and an exact-only continuation completed. No measurement heap was raised and no query was retried.

Successful-case timing gives ALL modes a 2-GiB Driver and two 512-MiB Executors. Exactly two reverse-order blocks (U F S L / L S F U), fresh JVM/query, same frozen physical input and settings, serial execution without backup/cleanup/build/parsing during either whole block. Data inodes and warm OS caches are reused; order reversal does not equalize caches or fully balance carryover. All 16 queries pass the common consumer; independent exact checks are reused once per mode, outside timing. Two samples per mode/query support descriptive ranges only, not significance or p95. Timers include the common validation consumer.

Query	Mode	Block 1 / 2 seconds	midpoint vs upstream
limit20	upstream	7.335341 / 7.705225	+0.00%
limit20	off	7.818479 / 7.580383	+2.38%
limit20	lightweight	7.554195 / 7.460936	-0.17%
limit20	streaming	9.625539 / 9.759117	+28.88%
full	upstream	80.374490 / 81.038505	+0.00%
full	off	80.288788 / 80.697623	-0.26%
full	lightweight	80.344297 / 80.508822	-0.35%
full	streaming	82.950126 / 82.899079	+2.75%

Streaming's LIMIT midpoint is +28.88% (about +2.17 s); full read is +2.75%. These negative timing costs are retained. Full-read sampled Driver heap midpoint is approximately 1093.73 MiB upstream / 539.50 MiB lightweight / 381.00 MiB streaming; sampled peaks can miss real peaks. Streaming common-live Driver writes increase by about 120 MiB in both query types. They include runtime/log work and are not exact connector-only bytes. All query event logs report zero Spark shuffle/spill; that does not exclude connector-local sort/replay. Default off has LIMIT +2.38% / full -0.26% midpoint observations, not proof of zero overhead. All raw read samples, CPU/GC/deserialization, per-JVM heap/RSS/I/O, correctness bindings and limitations are included. No throughput claim is made from this single-host experiment.

Validation and reproduction

• Normal JDK8 Spark3 and JDK17 Spark4.0/4.1 builds; Core 50 focused risk regressions and Spark3 23 SQL/metadata/concurrency regressions passed. Spark4.1 has 11 tests via normal Maven; Spark4.0 has 11 via direct ScalaTest on the actual normal reactor classpath, not a default UT40 Maven claim. Shared warm Maven dependencies were used with separate verified outputs.
• Automatic upstream CI run 43611962280: 18 successful, 1 skipped. The earlier run exposed missing generated option references; those were regenerated with the repository formatter, and both Core jobs now pass.
• Independent source/compatibility review completed. Source-only reproduction kit, commands, exact SQL, all results and resource limits reuse the original helper/controller and distinguish fixture SDK origin from the fixed comparison runtime. The assembled kit has not been rerun as a fresh-host package.

The controlled SELECT/LIMIT acceptance above is independent of MERGE acceptance. The production three-million-file exit137 incident remains unclassified and is not claimed fixed.