You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
With #34, an idle Redis worker waits on a marker with BZPOPMIN instead of polling. When a single Redis core saturates, the PR drains fewer jobs than polling did, because each job costs more round trips.
Measured on a 16-vCPU orb, Redis 7.4 pinned to one dedicated core, workers and producer pinned to other cores, 6 interleaved reps per variant (head 9b55a64):
Cell
main drained/s
PR drained/s
Round trips/job
100 workers, concurrency 1, 10k/s
8932
7145 (−20%)
4.0 → 5.9
32 workers, concurrency 10, 10k/s
9070
7459 (−18%)
100 workers, concurrency 10, 5k/s
4897
4992
Redis time per job: 102 µs on main, 138 µs on the PR. Round trips cost Redis about 14 µs each, more than the work inside the scripts.
Where the extra round trips go
A job wakes several workers. Per job, at saturation:
1.2 to 1.9 BZPOPMIN returns;
1.3 to 2.4 popFrom calls that find nothing;
up to 0.34 RELEASE_MARKER calls (32 workers, concurrency 10).
The push adds a ready marker, and the acquisition adds it again while jobs remain, so both often wake a worker for the same job.
Ideas, untested
Re-mark only when more jobs remain than workers already woken, or carry a count in the marker.
Let the acquisition script pop the next job for a worker woken by a marker, so a wake costs one round trip.
Drop the release round trip for a ready marker popped after the wait ended, if another marker already covers the queue.
Any change must keep the guarantee that a stored job wakes at least one idle worker, or is found at the next stalled check.
Activity