Skip to content

perf(redis): wake fewer workers per job #35

Open

Description

@RomainLanz

Problem

With #34, an idle Redis worker waits on a marker with BZPOPMIN instead of polling. When a single Redis core saturates, the PR drains fewer jobs than polling did, because each job costs more round trips.

Measured on a 16-vCPU orb, Redis 7.4 pinned to one dedicated core, workers and producer pinned to other cores, 6 interleaved reps per variant (head 9b55a64):

Cell main drained/s PR drained/s Round trips/job
100 workers, concurrency 1, 10k/s 8932 7145 (−20%) 4.0 → 5.9
32 workers, concurrency 10, 10k/s 9070 7459 (−18%)
100 workers, concurrency 10, 5k/s 4897 4992

Redis time per job: 102 µs on main, 138 µs on the PR. Round trips cost Redis about 14 µs each, more than the work inside the scripts.

Where the extra round trips go

A job wakes several workers. Per job, at saturation:

  • 1.2 to 1.9 BZPOPMIN returns;
  • 1.3 to 2.4 popFrom calls that find nothing;
  • up to 0.34 RELEASE_MARKER calls (32 workers, concurrency 10).

The push adds a ready marker, and the acquisition adds it again while jobs remain, so both often wake a worker for the same job.

Ideas, untested

  • Re-mark only when more jobs remain than workers already woken, or carry a count in the marker.
  • Let the acquisition script pop the next job for a worker woken by a marker, so a wake costs one round trip.
  • Drop the release round trip for a ready marker popped after the wait ended, if another marker already covers the queue.

Any change must keep the guarantee that a stored job wakes at least one idle worker, or is found at the next stalled check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions