Repository navigation
Conversation
jeffdupont
left a comment
There was a problem hiding this comment.
Thanks Paul, this is close. Judges run only after the shadow succeeds, outside asyncio.timeout, with the task's conversation id, and a judge failure never becomes the shadow's error. Two things have to change before it can be approved:
- The spec change has to land first. The spec as merged says no judge runs on a shadow request, so this PR changes behaviour the spec rules out. Once the spec change is up and agreed, I'll re-review against it.
- Public repo text. Please remove the ticket key from the PR title and the commit messages, and drop the
Ticket:andSpec change:lines from the body.
One more fix, on a line this diff doesn't touch: the ShadowTask docstring (shadow_requests.py:70) still says "with no flag evaluation". A judged task now evaluates the judges' flags when it runs. The Go and JS PRs both updated theirs to say it evaluates no flag except the shadow's judges.
Optional nits are inline. Three with no line to attach to: no test for the no-matching-handler case with judges attached; no assertion that the judge sees the task's conversation id (_Recorder already records it); and no positive check that B's handler receives judgeConfiguration (the old negative assertion was removed). Also, the body says 13 cases, but TestJudges has 11 methods, 15 cases after parametrize.
| result: dict[str, Any], | ||
| ) -> dict[str, JudgeResult]: | ||
| """Scores the shadow's response with its own judges, outside the shadow timeout.""" | ||
| response = _resolve_output_format_response( |
There was a problem hiding this comment.
Nit (optional): This call sits outside the try. It can't raise today, but if it ever did, run_shadow would raise, breaking its "never raises" promise, and a background shadow would end with an unretrieved exception. Moving it inside the try matches the JS PR.
There was a problem hiding this comment.
Moved the judge call inside the try in ea7e5f3, under the same conversation binding as the shadow call, so run_shadow keeps its never-raises promise. The output-format resolution now sits inside _run_judges's own try too, so a judge-side failure still never becomes the shadow's error.
| await sr._wait_for_shadows_for_testing() | ||
| assert len(judge.calls) == 1 | ||
|
|
||
| async def test_shadow_judge_event_is_tracked_under_the_shadow(self) -> None: |
There was a problem hiding this comment.
Nit (optional): This checks variationKey, shadowRequestId, primaryVariationKey, judgeConfigKey and runId on the judge event, plus isShadow through the filter, but not ownerKind or ownerId. The JS test asserts all five shadow fields.
There was a problem hiding this comment.
Added ownerKind and ownerId asserts in ea7e5f3, so the judge event test now covers all five shadow fields.
fac5f70 to
ea7e5f3
Compare
|
Thanks Jeff. Pushed ea7e5f3 with the rest of the review:
|
ea7e5f3 to
36e591f
Compare
jeffdupont
left a comment
There was a problem hiding this comment.
Thanks Paul. Everything from my last review is addressed, and the client fix in this latest commit matches the JS and Go PRs: shadow judges now never start a client, and the two new tests pin it.
The one thing blocking approval is the same as last time: the spec change has to land first. The spec as merged says no judge runs on a shadow request. When it goes up, it's worth stating that on a shadow any judge failure is logged and skipped while the others keep scoring, that resolving a shadow's judges never starts a client, and that judge_results is present when a judge scored.
Two more fixes, both inline. One optional nit: test_judge_failure_is_not_a_shadow_failure doesn't assert that the failure was logged, where the JS test asserts its warning. caplog would pin it.
| tool_handlers: Mapping[str, Callable[..., Any] | NativeTool] | None = None, | ||
| graph_key: str | None = None, | ||
| trajectory: str = "", | ||
| existing_client_only: bool = False, |
There was a problem hiding this comment.
Should-fix: This makes existing_client_only part of the public API of run_judges and extract_variation (the root API report changed), for something only the experimental shadow path uses. In JS the same flag stays internal, and in Go it's on an internal options struct. Could it stay private here too, for example an underscored keyword or a private helper the shadow module calls, so the core API doesn't grow for an experimental feature?
There was a problem hiding this comment.
Done. run_judges and extract_variation are back to their signatures on main, and the root API report has no diff. The shadow module now calls a private _run_judges with a resolver, passing _extract_existing_variation, which uses the client already set up and never starts one.
| assert judge.calls == [] | ||
| assert _judge_events(client) == [] | ||
|
|
||
| async def test_judge_failure_is_not_a_shadow_failure(self) -> None: |
There was a problem hiding this comment.
Should-fix: This uses a single judge, so nothing pins that one judge's failure leaves another judge's score. Both sibling PRs test that with two judges: one fails, and the other's score event is still sent. Python's code already does the right thing (judges.py, the per-judge except), so this is only a test, but it would catch a regression if run_judges ever started raising.
There was a problem hiding this comment.
Added test_one_judge_failing_leaves_the_others_scoring: two judges, the first one's provider raises, the second still scores and sends its event, judge_results holds only the second, and error stays unset. The failure test now asserts the error log with caplog too.
A shadow variation's judges now score its response after it succeeds, with each judge's own sampling rate. Scores are tracked with the shadow's track data (isShadow, the shadow variationKey, the served runId), and ShadowResult.judge_results carries them. The shadow timeout still covers only the shadow handler, and skip_judges applies to the served variation. Judges run inside run_shadow's error handling, so run_shadow never raises. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
36e591f to
08e63ae
Compare
Human
Shadow requests tell you whether a candidate model is as good as the live one, but today the shadow variation's judges are stripped, so a shadow run reports latency, tokens, and errors and no quality scores. This runs the shadow variation's judges on the shadow response, after it succeeds and off the request path, and tracks each score under the shadow variation with isShadow set.
Bot
_SHADOW_ONLY_KEYSno longer dropsjudgeConfiguration, so B's handler sees it exactly as A's does._runcalls corerun_judgeswith B's shadowtrack_dataas the base, A'suser_input, B's response throughjudge_textand B's trajectory. It runs under the task's conversation id and outside theasyncio.timeout, with the unwrapped handler, and inside_run'strysorun_shadownever raises.error.ShadowResultgainsjudge_results: dict[str, JudgeResult] | None, set only when a judge ran. The API report is regenerated.skip_judgesstill applies only to the served variation.run_shadowshare one code path.ShadowTaskstill pickles, now carrying the judge config, and its docstring says running it evaluates no flag except the shadow's judges.TestJudgesclass with 14 methods, 16 cases after parametrize (judge scores B, B's handler receivesjudgeConfiguration, no judge without a matching handler for B, judge sees the task's conversation id, runs after the caller returns, all shadow fields on the judge event and none on A's, sampling rates 0 and 1, no judge on error or timeout, judge failure is not a shadow failure, the timeout does not cover judges,judge_resultspresent or absent,skip_judges, deferred equals background, pickle). The default shadow fixture no longer carries a judge. 11 of the new cases fail on main.make test(4,190 passed, 18 skipped),make lint,make format-check,make typecheck,make api-report-check.🤖 Generated with Claude Code
Note
Overview
Shadow requests now run each candidate variation’s configured judges after a successful shadow generation, off the request path and outside the shadow
timeout_s, so quality metrics can be compared alongside latency and tokens.Shadow configs retain
judgeConfiguration(onlytoolsand nested shadow metadata are still stripped for the invoke). Scores are tracked on the usual judge events withisShadow: trueand the served run’srunId; failures in individual judges are logged and skipped without failing the shadow.ShadowResultgains optionaljudge_results;skip_judgesstill affects only the served call.Judge execution is factored through a shared
_run_judgeswith an injectable variation resolver. Shadow judges use_extract_existing_variationso they never lazy-init the LD client (e.g. aftershutdown()).Reviewed by Cursor Bugbot for commit 08e63ae. Bugbot is set up for automated code reviews on this repo. Configure here.