Repository navigation
Conversation
…nce per call, not once per chunk (refs ArcadeData#315) A vector column (a list of lists for an ARRAY_OF_FLOATS property) is laid out as a float32 matrix only when the property is declared, so the packer asks for the declared types, and it asked for every chunk of 1000 rows with a full read of schema:types. These tests count the schema reads per call (at the statement, on the handle's query), for insert_many, an insert_columns that falls back to rows, and an insert_arrow whose batches fall back to rows: one per call, where the old code made one per chunk. A generator that creates the type and a vector property on the first chunk and another property before the second must still see every chunk as columns, and two successive calls see a schema change made between them. Declared is unit-tested on a stub handle: it reads once, reads again after a statement ran on the handle, never reads when given its types, and asks again while the schema cannot be read. They fail on the current code (it reads once per chunk and has no statement stamp on the handle). Nothing is measured here. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…l, not once per chunk (refs ArcadeData#315) A vector column (a list of lists for an ARRAY_OF_FLOATS property) is laid out as a float32 matrix only when the property is declared, and the packer found that out with a full SELECT FROM schema:types (every type and property in one reply) for every chunk of 1000 rows of insert_many, and again for every batch of the row fallback of insert_columns, insert_arrow and insert_dataframe. Now one Declared belongs to one call: insert_many and insert_columns make it (columnar.declared_for) and every chunk shares it, and insert_table passes the types it already read to the row fallback of each batch. It is dropped when the call ends, so nothing is kept across calls. A schema change made while a call runs is still seen by the next chunk. The ingest itself never changes the schema (the image takes the type with getType, and only openCypher CREATE and MERGE make types or properties implicitly), so the one way is the caller's own code running while a generator of rows is read, and that is a statement on the handle: NativeDatabase counts the statements run through command() (_command_count), and Declared reads again when the count moved since it read. A query, a transaction call or a bulk crossing does not move it, so an ordinary load reads once. A read that fails is not kept, as before a chunk asked again. _insert_chunk takes the call's Declared before the chunk, because the ingest-integrity mutation tests wrap it as (type_name, *lead, chunk). The public insert_many is unchanged. Not measured yet: the speed claim waits for a timed A/B of a candidate wheel against 26.11.1.post2. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Collaborator
Author
|
The timed A/B of the post3 candidate against the released post2 (results on #315) shows no measurable change for this PR: the native |
This was referenced Nov 13, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
On the native wheel, a vector column (a list of lists loading into an
ARRAY_OF_FLOATSproperty) crosses as a float32 matrix only when the property is declared, so the packer reads the declared types. It did that with a fullSELECT FROM schema:types(every type and property in one reply) for every chunk of 1000 rows ofinsert_many, and again for every batch of the row fallback ofinsert_columns,insert_arrowandinsert_dataframe. I changed it to read once per call and share the answer between the chunks.Reads of
schema:typesper call when a vector column needs the declared types, as read from the code:insert_many, 3,500 rows,commit_every=1000insert_columnsthat falls back to rows, 2,500 rows,commit_every=1000insert_arroworinsert_dataframewhose batches fall back to rowsHow a stale answer is excluded:
columnar.Declaredbelongs to one call.insert_manyandinsert_columnsmake it, andinsert_tablehands the types it read at the start to the row fallback of each batch. It is a local of the call, so nothing is kept across calls: a schema change made between two calls is seen by the second.getType(a missing type is an error) and the bulk paths declare no property. The engine makes types and properties implicitly only for openCypherCREATEandMERGE. So the one way the schema changes inside a call is the caller's own code running while a generator of rows is read.NativeDatabasenow counts the statements it runs throughcommand()(_command_count, one integer add in_exec), andDeclaredreads again when the count moved since its last read. A query, a transaction call or a bulk crossing does not move it, so an ordinary load reads once.Everything else is as before. A dict of known types (the DataFrame path) is still used as it is, and a read that fails is not kept, so the next column asks again, as the next chunk did.
_insert_chunktakes the call'sDeclaredbefore the chunk because the ingest-integrity mutation tests wrap it as(type_name, *lead, chunk). The publicinsert_manyis unchanged.Related issues
Part of #315.
Performance changes
Not measured yet: the speed claim waits for a timed A/B of a candidate wheel against 26.11.1.post2. Fewer reads is what the code does; how much time it saves is not known.
How I verified it
The tests came first, in their own commit, and extend
tests/test_native_columnar_insert.pyandtests/test_native_columnar_pack.py. They count the schema reads at the statement (the handle'squerywithschema:types), never time:insert_many,insert_columnswith a row fallback andinsert_arrowwith a column that cannot cross each read once per call, and store what the JSON path stores.Declaredon a stub handle: one read, a read again after a statement, never a read for a given dict, a failed read asked again.Locally I ran only light checks: compile, pyflakes, black, isort, ruff and bandit on the changed files,
uvx pre-commit run --fileson them, and the fourDeclaredtests against the old and the newcolumnar.pywith a stub handle (they fail on the old one and pass on the new one). The integration tests need the native library, so they run in CI only, and I have not seen them run yet.Docs
None. No public API or behavior changes.
Checklist
Declaredtests do; the integration tests wait for CI)uvx pre-commit run --files <changed files>passes🤖 Generated with Claude Code