[ddeboer]

Typesense 30.2 permanently loses an async_reference written while the referenced collection is being imported. Until now the docs said to run the indexer twice on a first build. That fails for REBUILD_MODE=blue-green, because every blue/green run is a first build.

Every batch import (upsert or update) now goes through a lock shared by its join component. Any number of imports into one collection run at once. An import into another collection of the component waits until none is in flight. Waiters go in arrival order, and all waiters of the admitted collection go in together, so a busy collection cannot starve the others. A type outside any join never waits.

Where the lock lives. The issue proposed searchIndexWriter, but the router only sees a stream of documents per type. The batches are cut inside @lde/search-typesense (documents.ts), so the router cannot tell when an import is in flight. Locking a whole write per type would deadlock a write call that feeds two collections of one component. So the lock wraps the one function that sends an import to Typesense. Each writer looks the lock up from its schema’s join component, the way joinGraph caches its graph per schema. No new option or interface. The writers of a component share the lock as long as they get the same SearchSchema, which searchIndexWriter always passes.

Why a local lock. I checked the dependency tree first. async-lock is a keyed mutex: exclusive per key, so imports into one collection could not overlap. Readers-writer locks such as @esfx/async-readerwriterlock have one shared group and one exclusive one. They fit a component of two collections only by making one of them exclusive with itself, and LOL’s component has three. Nothing I found gives concurrency within a group and exclusion between any number of groups, so GroupLock is about 50 local lines with unit tests.

Test. The joins end-to-end test now runs the indexer once, on 900 datasets, 900 works and 50 publishers with a batch size of 600. That makes datasets land before their publishers exist and the flush land all three collections at once. A proxied client checks after every import into a referenced collection, before the lock lets the next import start, that every referrer pointing at the just-imported documents already joins to them. That confirms Typesense finishes back-filling before the import call returns. With the lock disabled, the test failed 5 runs out of 5 (“publishers: 600 of 902 datasets joined”). With the lock, it passed 6 out of 6.

Docs. The “run the indexer twice” warning is gone from search-typesense.md and search.md. New ADR 39, “Serialise imports across a join component’s collections”, amends ADR 19’s limitation. The duplicate ADR 31 (local files through the data dump path) is now 38, with its three links updated. The issue asked for 35 and 34, but both had been taken since (34–37 exist on main), so I used the next free numbers.

Still open: a Linked Open Limburg run timed before and after this change.

Part of #1023