Skip to content

Native columnar ingest: the declared property types are read once per call, not once per chunk (refs #315) - #386

Draft
tae898 wants to merge 2 commits into
ArcadeData:mainfrom
humemai:perf/315-native-declared-once
Draft

tae898 wants to merge 2 commits into
ArcadeData:mainfrom
humemai:perf/315-native-declared-once

Conversation

@tae898

@tae898 tae898 commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator

What does this PR do?

On the native wheel, a vector column (a list of lists loading into an ARRAY_OF_FLOATS property) crosses as a float32 matrix only when the property is declared, so the packer reads the declared types. It did that with a full SELECT FROM schema:types (every type and property in one reply) for every chunk of 1000 rows of insert_many, and again for every batch of the row fallback of insert_columns, insert_arrow and insert_dataframe. I changed it to read once per call and share the answer between the chunks.

Reads of schema:types per call when a vector column needs the declared types, as read from the code:

Call Before After
insert_many, 3,500 rows, commit_every=1000 4 (one per chunk) 1
insert_columns that falls back to rows, 2,500 rows, commit_every=1000 4 (one for the columnar attempt, one per chunk) 1
insert_arrow or insert_dataframe whose batches fall back to rows 1 plus one per chunk of each batch 1
a call where no column needs the declared types 0 or 1 unchanged

How a stale answer is excluded:

  • One columnar.Declared belongs to one call. insert_many and insert_columns make it, and insert_table hands the types it read at the start to the row fallback of each batch. It is a local of the call, so nothing is kept across calls: a schema change made between two calls is seen by the second.
  • The ingest never changes the schema itself. The image takes the type with getType (a missing type is an error) and the bulk paths declare no property. The engine makes types and properties implicitly only for openCypher CREATE and MERGE. So the one way the schema changes inside a call is the caller's own code running while a generator of rows is read.
  • That is a statement on the handle. NativeDatabase now counts the statements it runs through command() (_command_count, one integer add in _exec), and Declared reads again when the count moved since its last read. A query, a transaction call or a bulk crossing does not move it, so an ordinary load reads once.
  • Not covered: a schema change made on another handle, thread or process while a call runs. The old per-chunk read could see it, as a race. I think that is fine for a bulk load, but it is a difference.

Everything else is as before. A dict of known types (the DataFrame path) is still used as it is, and a read that fails is not kept, so the next column asks again, as the next chunk did. _insert_chunk takes the call's Declared before the chunk because the ingest-integrity mutation tests wrap it as (type_name, *lead, chunk). The public insert_many is unchanged.

Related issues

Part of #315.

Performance changes

Not measured yet: the speed claim waits for a timed A/B of a candidate wheel against 26.11.1.post2. Fewer reads is what the code does; how much time it saves is not known.

How I verified it

The tests came first, in their own commit, and extend tests/test_native_columnar_insert.py and tests/test_native_columnar_pack.py. They count the schema reads at the statement (the handle's query with schema:types), never time:

  • insert_many, insert_columns with a row fallback and insert_arrow with a column that cannot cross each read once per call, and store what the JSON path stores.
  • A generator that creates the type and a vector property while the first chunk is read, and another property before the second, still sends every chunk as columns and reads twice, not three times.
  • Two successive calls each read the schema, and the second sees a property made between them.
  • Declared on a stub handle: one read, a read again after a statement, never a read for a given dict, a failed read asked again.

Locally I ran only light checks: compile, pyflakes, black, isort, ruff and bandit on the changed files, uvx pre-commit run --files on them, and the four Declared tests against the old and the new columnar.py with a stub handle (they fail on the old one and pass on the new one). The integration tests need the native library, so they run in CI only, and I have not seen them run yet.

Docs

None. No public API or behavior changes.

Checklist

  • A regression test fails on the old code and passes on the new (the Declared tests do; the integration tests wait for CI)
  • The test suite that the change can affect passes (waits for the native CI run)
  • uvx pre-commit run --files <changed files> passes

🤖 Generated with Claude Code

tae898 and others added 2 commits November 14, 2026 03:03
…nce per call, not once per chunk (refs ArcadeData#315)

A vector column (a list of lists for an ARRAY_OF_FLOATS property) is laid out as a float32 matrix only when the property is declared, so the packer asks for the declared types, and it asked for every chunk of 1000 rows with a full read of schema:types.

These tests count the schema reads per call (at the statement, on the handle's query), for insert_many, an insert_columns that falls back to rows, and an insert_arrow whose batches fall back to rows: one per call, where the old code made one per chunk. A generator that creates the type and a vector property on the first chunk and another property before the second must still see every chunk as columns, and two successive calls see a schema change made between them. Declared is unit-tested on a stub handle: it reads once, reads again after a statement ran on the handle, never reads when given its types, and asks again while the schema cannot be read.

They fail on the current code (it reads once per chunk and has no statement stamp on the handle). Nothing is measured here.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…l, not once per chunk (refs ArcadeData#315)

A vector column (a list of lists for an ARRAY_OF_FLOATS property) is laid out as a float32 matrix only when the property is declared, and the packer found that out with a full SELECT FROM schema:types (every type and property in one reply) for every chunk of 1000 rows of insert_many, and again for every batch of the row fallback of insert_columns, insert_arrow and insert_dataframe.

Now one Declared belongs to one call: insert_many and insert_columns make it (columnar.declared_for) and every chunk shares it, and insert_table passes the types it already read to the row fallback of each batch. It is dropped when the call ends, so nothing is kept across calls.

A schema change made while a call runs is still seen by the next chunk. The ingest itself never changes the schema (the image takes the type with getType, and only openCypher CREATE and MERGE make types or properties implicitly), so the one way is the caller's own code running while a generator of rows is read, and that is a statement on the handle: NativeDatabase counts the statements run through command() (_command_count), and Declared reads again when the count moved since it read. A query, a transaction call or a bulk crossing does not move it, so an ordinary load reads once. A read that fails is not kept, as before a chunk asked again.

_insert_chunk takes the call's Declared before the chunk, because the ingest-integrity mutation tests wrap it as (type_name, *lead, chunk). The public insert_many is unchanged.

Not measured yet: the speed claim waits for a timed A/B of a candidate wheel against 26.11.1.post2.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@tae898 tae898 added python Pull requests that update python code performance Speed or memory of the bindings, with a measurement native The arcadedb package (native library build) labels Nov 13, 2026
@tae898 tae898 self-assigned this Nov 13, 2026
@tae898

tae898 commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator Author

The timed A/B of the post3 candidate against the released post2 (results on #315) shows no measurable change for this PR: the native insert_columns figure with a vector column in 100 chunks has B/A 1.000 with an interval of 0.991 to 1.008. The schema reads per call drop from several to one, but the figure does not move on a one-type schema, and the change adds an integer to every native command. I am leaving it out of the post3 candidate for now. It would need a figure that shows a gain, for example a schema with many types, before it goes in.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

native The arcadedb package (native library build) performance Speed or memory of the bindings, with a measurement python Pull requests that update python code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant