Skip to content

test(computers): two recording tests wait for what they check instead of racing it - #853

Merged
earakely-scale merged 2 commits into
mainfrom
edgararakelyan/flaky-recording-tests
Nov 20, 2026
Merged

earakely-scale merged 2 commits into
mainfrom
edgararakelyan/flaky-recording-tests

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

Two unit tests failed on #844's CI runs and passed on rerun. Neither failure came from that change; each was a race
in the test.

  • test_browser_recording.py::test_the_video_follows_an_agents_tab_a_popup_it_opens_and_goes_back_when_that_closes
    kept the popup (red) open for a fixed 0.3 s. On a loaded runner a frame decode can take longer, so the 10 fps video
    took no red frame and the colour check failed.
  • test_computer_recording_annotate.py::test_an_input_goes_ahead_when_its_mark_cant_be_made made the fake's next
    display read fail, expecting the scroll's mark to be the one that reads it. The recording's own capture loop also
    reads the display at 10 fps and could take the failure first, so the scroll was marked.

What changes

  • The browser test waits until the video has encoded a frame of each colour (green, yellow, red, then yellow again)
    before it moves on, with a 10 s limit, instead of fixed sleeps. It notes the frames by wrapping the stream's
    _latest, which the video calls for each frame. The video takes a frame only once it has encoded the one before,
    so a colour is encoded once the frame after its first has been taken.
  • The annotate test makes the mark itself fail (annotate.input_marks raises) instead of the shared display read. It
    still goes through the same best-effort except in ComputerRuntime._annotate.

Tests only.

How it was checked

Each race made to happen on purpose, by running both versions of each test with a slowdown that a loaded runner causes:

Test Slowdown main This branch
popup video each frame decode takes 0.35 s fails: no red frame, as on CI passes
mark can't be made the pointer read takes 0.25 s fails: the scroll is marked, as on CI passes
popup video, the first commit's wait (taken, not encoded) each frame's conversion takes 0.3 s fails 4 of 4: the last yellow frame isn't encoded passes 4 of 4

Both files pass (58 tests), and each test passed 40 times in a row under twelve CPU busy loops.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge.

Summary

Two recording tests replace timing races with checks tied to the work they need.

  • Browser recording tests wait for the frames they check.
  • The annotation test makes mark creation fail directly.

The earlier last-frame finding is fixed: the next sample cannot arrive until the previous frame finishes encoding. No new actionable issues were found. Tests were not run during this review.

Reviews (5) · Last reviewed commit: "test(computers): the popup test waits un..." · Reviewed by Greptile

… of racing it

The browser recording test that follows a popup waits until the video has taken a frame of each tab's colour, not a
fixed 0.3 s, which a slow frame decode on a loaded runner outlasted, so no red frame was recorded. The annotated
recording test whose mark can't be made fails the mark itself rather than the display read, which the recording's
own captures could take first, leaving the scroll marked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 20, 2026 22:37
…ot just taken

The video takes a frame only once it has encoded the one before, so a frame taken after a colour's first proves that
one encoded; ending the recording sooner could cancel its last frame mid-conversion.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale merged commit 38f759c into main Nov 20, 2026
38 checks passed
@earakely-scale
earakely-scale deleted the edgararakelyan/flaky-recording-tests branch November 20, 2026 22:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant