Skip to content

feat(shadow-requests): run judges on shadow responses - #302

Open
ld-paul wants to merge 1 commit into
mainfrom
aic-3882-shadow-judges
Open

ld-paul wants to merge 1 commit into
mainfrom
aic-3882-shadow-judges

Conversation

@ld-paul

@ld-paul ld-paul commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Human

Shadow requests tell you whether a candidate model is as good as the live one, but today the shadow variation's judges are stripped, so a shadow run reports latency, tokens, and errors and no quality scores. This runs the shadow variation's judges on the shadow response, after it succeeds and off the request path, and tracks each score under the shadow variation with isShadow set.

Bot

  • _SHADOW_ONLY_KEYS no longer drops judgeConfiguration, so B's handler sees it exactly as A's does.
  • After B succeeds, _run calls core run_judges with B's shadow track_data as the base, A's user_input, B's response through judge_text and B's trajectory. It runs under the task's conversation id and outside the asyncio.timeout, with the unwrapped handler, and inside _run's try so run_shadow never raises.
  • Judges only run on success. A judge failure is logged and never sets error.
  • ShadowResult gains judge_results: dict[str, JudgeResult] | None, set only when a judge ran. The API report is regenerated.
  • skip_judges still applies only to the served variation.
  • Background mode and run_shadow share one code path. ShadowTask still pickles, now carrying the judge config, and its docstring says running it evaluates no flag except the shadow's judges.
  • Tests: a new TestJudges class with 14 methods, 16 cases after parametrize (judge scores B, B's handler receives judgeConfiguration, no judge without a matching handler for B, judge sees the task's conversation id, runs after the caller returns, all shadow fields on the judge event and none on A's, sampling rates 0 and 1, no judge on error or timeout, judge failure is not a shadow failure, the timeout does not cover judges, judge_results present or absent, skip_judges, deferred equals background, pickle). The default shadow fixture no longer carries a judge. 11 of the new cases fail on main.
  • Checks: make test (4,190 passed, 18 skipped), make lint, make format-check, make typecheck, make api-report-check.

🤖 Generated with Claude Code


Note

Overview
Shadow requests now run each candidate variation’s configured judges after a successful shadow generation, off the request path and outside the shadow timeout_s, so quality metrics can be compared alongside latency and tokens.

Shadow configs retain judgeConfiguration (only tools and nested shadow metadata are still stripped for the invoke). Scores are tracked on the usual judge events with isShadow: true and the served run’s runId; failures in individual judges are logged and skipped without failing the shadow. ShadowResult gains optional judge_results; skip_judges still affects only the served call.

Judge execution is factored through a shared _run_judges with an injectable variation resolver. Shadow judges use _extract_existing_variation so they never lazy-init the LD client (e.g. after shutdown()).

Reviewed by Cursor Bugbot for commit 08e63ae. Bugbot is set up for automated code reviews on this repo. Configure here.

@ld-paul
ld-paul requested a review from a team as a code owner November 13, 2026 21:53

@jeffdupont jeffdupont left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Paul, this is close. Judges run only after the shadow succeeds, outside asyncio.timeout, with the task's conversation id, and a judge failure never becomes the shadow's error. Two things have to change before it can be approved:

  1. The spec change has to land first. The spec as merged says no judge runs on a shadow request, so this PR changes behaviour the spec rules out. Once the spec change is up and agreed, I'll re-review against it.
  2. Public repo text. Please remove the ticket key from the PR title and the commit messages, and drop the Ticket: and Spec change: lines from the body.

One more fix, on a line this diff doesn't touch: the ShadowTask docstring (shadow_requests.py:70) still says "with no flag evaluation". A judged task now evaluates the judges' flags when it runs. The Go and JS PRs both updated theirs to say it evaluates no flag except the shadow's judges.

Optional nits are inline. Three with no line to attach to: no test for the no-matching-handler case with judges attached; no assertion that the judge sees the task's conversation id (_Recorder already records it); and no positive check that B's handler receives judgeConfiguration (the old negative assertion was removed). Also, the body says 13 cases, but TestJudges has 11 methods, 15 cases after parametrize.

result: dict[str, Any],
) -> dict[str, JudgeResult]:
"""Scores the shadow's response with its own judges, outside the shadow timeout."""
response = _resolve_output_format_response(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit (optional): This call sits outside the try. It can't raise today, but if it ever did, run_shadow would raise, breaking its "never raises" promise, and a background shadow would end with an unretrieved exception. Moving it inside the try matches the JS PR.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved the judge call inside the try in ea7e5f3, under the same conversation binding as the shadow call, so run_shadow keeps its never-raises promise. The output-format resolution now sits inside _run_judges's own try too, so a judge-side failure still never becomes the shadow's error.

await sr._wait_for_shadows_for_testing()
assert len(judge.calls) == 1

async def test_shadow_judge_event_is_tracked_under_the_shadow(self) -> None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit (optional): This checks variationKey, shadowRequestId, primaryVariationKey, judgeConfigKey and runId on the judge event, plus isShadow through the filter, but not ownerKind or ownerId. The JS test asserts all five shadow fields.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added ownerKind and ownerId asserts in ea7e5f3, so the judge event test now covers all five shadow fields.

@ld-paul
ld-paul force-pushed the aic-3882-shadow-judges branch from fac5f70 to ea7e5f3 Compare November 16, 2026 17:00
@ld-paul ld-paul changed the title feat(shadow-requests): run judges on shadow responses (AIC-3882) feat(shadow-requests): run judges on shadow responses Nov 16, 2026
@ld-paul

ld-paul commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Thanks Jeff. Pushed ea7e5f3 with the rest of the review:

  • The ShadowTask docstring now says running it evaluates no flag except the shadow's judges, matching the Go and JS wording.
  • Three new tests: no judge runs when judges are attached but no handler matches B, the judge sees the task's conversation id (run under a different worker conversation), and B's handler receives judgeConfiguration.
  • The body's count is fixed: TestJudges is now 14 methods, 16 cases after parametrize.
  • The title, commit message, and body are cleaned up.

@ld-paul
ld-paul force-pushed the aic-3882-shadow-judges branch from ea7e5f3 to 36e591f Compare November 16, 2026 17:41

@jeffdupont jeffdupont left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Paul. Everything from my last review is addressed, and the client fix in this latest commit matches the JS and Go PRs: shadow judges now never start a client, and the two new tests pin it.

The one thing blocking approval is the same as last time: the spec change has to land first. The spec as merged says no judge runs on a shadow request. When it goes up, it's worth stating that on a shadow any judge failure is logged and skipped while the others keep scoring, that resolving a shadow's judges never starts a client, and that judge_results is present when a judge scored.

Two more fixes, both inline. One optional nit: test_judge_failure_is_not_a_shadow_failure doesn't assert that the failure was logged, where the JS test asserts its warning. caplog would pin it.

tool_handlers: Mapping[str, Callable[..., Any] | NativeTool] | None = None,
graph_key: str | None = None,
trajectory: str = "",
existing_client_only: bool = False,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should-fix: This makes existing_client_only part of the public API of run_judges and extract_variation (the root API report changed), for something only the experimental shadow path uses. In JS the same flag stays internal, and in Go it's on an internal options struct. Could it stay private here too, for example an underscored keyword or a private helper the shadow module calls, so the core API doesn't grow for an experimental feature?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. run_judges and extract_variation are back to their signatures on main, and the root API report has no diff. The shadow module now calls a private _run_judges with a resolver, passing _extract_existing_variation, which uses the client already set up and never starts one.

assert judge.calls == []
assert _judge_events(client) == []

async def test_judge_failure_is_not_a_shadow_failure(self) -> None:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should-fix: This uses a single judge, so nothing pins that one judge's failure leaves another judge's score. Both sibling PRs test that with two judges: one fails, and the other's score event is still sent. Python's code already does the right thing (judges.py, the per-judge except), so this is only a test, but it would catch a regression if run_judges ever started raising.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added test_one_judge_failing_leaves_the_others_scoring: two judges, the first one's provider raises, the second still scores and sends its event, judge_results holds only the second, and error stays unset. The failure test now asserts the error log with caplog too.

A shadow variation's judges now score its response after it succeeds,
with each judge's own sampling rate. Scores are tracked with the shadow's
track data (isShadow, the shadow variationKey, the served runId), and
ShadowResult.judge_results carries them. The shadow timeout still covers
only the shadow handler, and skip_judges applies to the served variation.
Judges run inside run_shadow's error handling, so run_shadow never raises.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@ld-paul
ld-paul force-pushed the aic-3882-shadow-judges branch from 36e591f to 08e63ae Compare November 17, 2026 00:59
@ld-paul
ld-paul requested a review from jeffdupont November 17, 2026 01:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants