[tae898]

What does this PR do?

On the JVM wheel, GraphBatch.create_vertices sent every chunk of rows as one JSON text (VertexBatcher.createVerticesJson): Python wrote the text, the engine parsed it into a JSONObject per row, a map per row was built from it, and the created RIDs came back as one ;-joined text that Python split. GraphBatch.new_edges with properties did the same through EdgeBatcher.newEdgesJson, and Python first built one dict per edge to carry the endpoints beside the properties.

Now rows that are all dicts with the same keys in the same order, each property holding one kind of int, float, bool or str (with None for a null), plus the float vectors that already crossed as a FloatVectors.Matrix, cross as one typed column per property, in the layout of DocumentBatcher.insertColumns (long[], double[], boolean[], StringColumn, with a boolean[] of null rows for a column that has a null). The new bridge methods are VertexBatcher.createVerticesColumns and EdgeBatcher.newEdgesColumns. They make the same engine call the text route made (GraphBatch.createVertices(type, Object[][]) and newEdge(src, type, dst, props...)), with the same name and value per property, so the batch settings (WAL, commit size, parallel flush, retries) apply as before. The RIDs of the vertices come back as one long[] of [bucket id, position] pairs, which Python turns into the same "#12:345" texts, in the same order, as before. The new edge call keeps the RIDs as the two joined texts of newEdgesJoined, which is one bulk copy each.

What keeps the old routes, so no input changes meaning: rows that are None or have other keys or another key order, a property that holds a list, tuple, dict, date-time, date, Decimal or bytes, a column of mixed kinds (an int beside a float or a bool), a numpy scalar, a subclass, an integer beyond 64 bits, a NaN, an infinity, a subnormal, a lone surrogate, and an edge property named _src or _dst. Those go to the JSON text route or to the property-matrix route exactly as they did, per chunk for vertices and for the whole call for edges (every edge chunk is built before the first crosses, so a chunk the typed route cannot carry starts the old route over with nothing buffered). Edges whose property dicts are all empty or None now use newEdgesJoined, the call for edges without properties, which stores the same edges.

One stored type is kept on purpose: the engine's JSON parser turns a number token that fits an int into an Integer, so the text route stored a small Python int of an undeclared property as INTEGER, while the per-element paths and insert_columns store LONG. The typed route boxes a long that fits an int as an Integer too, so the stored class does not change with the route. Whether the two should be unified is a separate decision.

Related issues

Refs #388 and #315.

Performance changes

No timed result yet: this is rank 9 of #388, and the A/B runs on the laptop timed lane. The figure to measure:

• Operation: GraphBatch.create_vertices(type, rows) and GraphBatch.new_edges(sources, type, destinations, properties) on the JVM wheel (arcadedb-jvm), one call per chunk as the benchmark lanes make them: rows of {id int, name str, age int, city str} at 5,000 rows per call (the l2 graph loads) with new_edges of [{"since": int}] at 5,000 per call; rows of {pid int, views int, embedding Java float[64]} in one call of 50,000 and of 500,000 rows (the e2 products, whose new_edges has no properties and is unchanged); rows of {vid int, embedding Java float[dim]} at 10,000 per call (the l3d loads). Smaller sizes (1 row, 100 rows) show the fixed cost per call.
• Unit of one operation: one call, reported per row (and per edge). Latency, peak RSS and anon growth are all of interest.
• Arms: the released 26.11.1.post3 wheel against a candidate wheel built from this branch, ABBA, with an A/A control. The 'no change' control is new_edges without properties and create_vertices(type, count), which this change does not touch, and create_vertices of rows that keep the text route (a list-valued property).
• Parity digest: the stored records of the load (property names in order, the Java class and the text of every value, the RIDs returned) must be equal between the arms; tests/test_graph_batch_typed_jvm.py holds the same comparison on a mix of int, float, bool and str columns with nulls, 32-bit and 64-bit boundaries, 17-digit doubles, -0.0, empty and 1,000-character strings, non-ASCII, vectors, an empty batch, a batch of one and a load over the chunk boundary.

What the change allocates, per chunk of n rows and k scalar properties: one temporary list of n references per property, one numpy array and its long[] or double[] copy (8 bytes per value; n bytes for a bool), a StringColumn for a str property (the UTF-8 bytes and 4 * (n + 1) bytes of offsets, as insert_columns makes), a boolean[] of n for a property with a null, and n temporary key tuples for the same-keys check. The JSON text (one string of the whole chunk, then its UTF-8 copy), the parsed JSONArray with a JSONObject and a map per row, and the joined RID text are no longer allocated. The Object[n][2k] matrix the engine takes is built as before. The returned RIDs are a long[2 * n] (16 bytes per vertex) before the list of n texts. For edges, the columns of every chunk of the call are held until it ends (about 8 bytes per numeric value of the call), where the text route held one dict per edge for the whole call.

Docs

docs/modules/ROOT/pages/how-to/bulk-loads.adoc (the paragraph on the property values of create_vertices and new_edges) and the docstrings of create_vertices say which rows cross as typed columns.

How I checked it

The tests are in the first commit and the change in the second, so a run of the first commit shows them failing on the old code.

• Red, on the tests only (commit 4c84313): https://github.com/ArcadeData/arcadedb-python/actions/runs/43566990576. The test jobs fail; on linux/amd64 with Python 3.12 it is 3039 passed and 24 failed, and the failures include tests of test_graph_batch_typed_jvm.py and test_graph_batch_vector_crossings_jvm.py.
• Green, on the change (commit 0af5e31): Test JVM run 43569512102, linux/amd64 with Python 3.12: 3060 passed and 3 failed. Two failures were bugs in my new tests (a read-back of @rid through SQL, and the sort key of the stored-record helper for an edge without an id), fixed in 0854265. The third, tests/test_import_database.py::test_import_database_xml_vertices ('FAIL' == 'OK', an engine IMPORT DATABASE of a local XML file), is not from this change: it also failed on the red run above, which has no production change, and on the Test JVM run of Native: GraphBatch.create_vertices sends columns and reads the RIDs from an int64 buffer #390, a native-only branch. A run on 0854265 is still to come.

tests/test_graph_batch_typed_jvm.py (JVM wheel only, listed in NATIVE_NOT_APPLICABLE): crossing counts (one createVerticesColumns or newEdgesColumns call per chunk, none of the JSON calls, no property-matrix fallback), the returned RIDs against SELECT @rid in row order and against the text route, parity of the stored records (typed route against the text route switched off) for vertices, vertices with vectors, a batch of one, a load over the chunk boundary and edges, and a test for every shape that keeps the text or matrix route (twelve vertex shapes, five edge shapes, a chunk that is not carried among carried ones, and the whole call going the old way for edges). The expectations of tests/test_graph_batch_vector_crossings_jvm.py move from createVerticesJson to createVerticesColumns, because the shapes there are now carried. uvx pre-commit run --files <changed files> and Bandit as CI runs it are clean. No native build is involved: the change is the bridge jar and graph_batch.py, and the native wheel has its own GraphBatch.

Checklist

• A regression test fails on the old code and passes on the new
• The test suite that the change can affect passes
• uvx pre-commit run --files <changed files> passes

🤖 Generated with Claude Code