Skip to content

fix(search-typesense): keep the rebuild lock alive while a run is open - #1080

Open
ddeboer wants to merge 2 commits into
mainfrom
fix/rebuild-lock-heartbeat
Open

ddeboer wants to merge 2 commits into
mainfrom
fix/rebuild-lock-heartbeat

Conversation

@ddeboer

@ddeboer ddeboer commented Nov 14, 2026

Copy link
Copy Markdown
Member

The rebuild lock recorded acquired_at once and never refreshed it. A healthy run that took longer than lockTtlMs lost its lock, so a second run could start alongside it, commit before it, and then have its newer data swapped out by the older run’s late commit. The older run’s release then deleted the newer run’s lock as well.

This change does what the issue proposes:

  • Heartbeat. While a run is open, its lock is refreshed every lockTtlMs / 3. The timer is unref’d, stops on release, and stops once the lock belongs to someone else. A refresh that fails is retried on the next beat. Another run can now only take the lock over once its holder has died, so the TTL no longer caps how long a rebuild may take. There is no maximum runtime.
  • Owner token. The lock document now records a random owner per acquisition. release deletes the lock only while this run still owns it, and commit checks ownership before going live: blue/green does not swap the alias and in-place does not sweep. A run that lost its lock rejects its commit, and the pipeline then aborts it.

Internally, acquireLock now returns a HeldLock (assertHeld, release) in place of a boolean, and the writers get that lock from openLockedRun. None of this is exported, so the public API is unchanged. I wrote the port contract into the RebuildAlreadyRunning JSDoc and the “Rebuild requests” section of the @lde/search docs, and updated the concurrency section of the search-typesense docs.

Older lock documents have no owner. They are still reclaimed once stale, and no run will delete one.

Fixes #968

A run now refreshes its lock every third of lockTtlMs, so the lock is only
reclaimed once its holder died, however long the run takes. The lock records
which run holds it: a run that lost its lock refuses to commit (blue/green
does not swap the alias, in-place does not sweep), and releasing never
deletes another run’s lock.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Keep the rebuild lock alive while a run is open

1 participant