Repository navigation
feat(prompt_agent): an opt-in gate on user-sim messages - #597
kaihackney-scale wants to merge 18 commits into
Conversation
prompt_agent's new user_sim_gate checks each LLM user-sim message before the agent receives it: phrase patterns for the assistant's side of a conversation, and optionally a small judge model that decides only whose voice the message is in. A message that fails is held back and the user-sim is asked to rewrite it, up to max_rewrites times; on_fail then fails the step, ends the conversation, or sends it anyway. Every verdict is recorded in context.metadata["user_sim_gate"]. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e its cutoffs
`user_sim_gate={}` now enables the gate with its defaults; only None leaves it off. The patterns' length and
count cutoffs, and how much of a judge's reason and error a verdict keeps, are named class constants.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
🤖 Automated Claude routine — Env Pod PR Context Review. This is a bot, not Edgar Arakelyan; not a human review. Context from outside this diff, in case it's useful: there's a second open PR adding a step type that subclasses
So in either merge order, an Related, same direction: #567 addresses the same failure mode from the prompt side — Generated by Claude Code |
…te where it would be ignored The check-and-rewrite loop moves to user_sim_gate.run_gate, with a rewrite callback, so a step type that runs its own conversation can apply the gate; prompt_agent's user side now calls it. A PromptAgentTaskStep subclass that replaces prompt_agent's conversation refuses user_sim_gate at construction, instead of accepting and ignoring it, unless it sets USER_SIM_GATE_SUPPORTED = True. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main reworked prompt_agent's user side (the built-in user sim, end_after). The gate keeps its history wrapper in ask(), main's end_after short-circuit opens _answer(), and the gate still checks peer user-sim messages. A gate set with the built-in user sim is warned about, as with a non-user-sim peer, since those messages aren't checked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…p errors fall back to the patterns - AlwaysOnTaskStep.from_dict passes user_sim_gate on, so a saved step is refused as one built in code is, rather than running with the setting dropped. - A judge whose setup fails (its key or endpoint) no longer ends the conversation: like a failed judge call, the patterns' verdict stands and the error is recorded on each verdict. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ps checking The fallback rebuilt the gate without its judge, which a judge-only gate (patterns off) refuses, so its setup error ended the conversation. The gate's check now takes judge_unavailable instead: the judge isn't called, the patterns' verdict stands (a pass with patterns off), and the reason is recorded on the verdict, as for a judge call that fails. run_gate passes it only when set, so a check override with the old signature still works. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…agent and always_on prompt_agent applies user_sim_gate to the user deploy_user_sim_agent deployed: a held-back answer is rewritten in its session and parsed as its own answer, and the read's record keeps the first answer and lists each rewrite. always_on applies it at each read before an answer is held: one that still fails is never delivered and ends the run (user_sim_invalid; an error with on_fail fail). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ite counts toward max_replies A conversation the gate cut off with on_fail fail now records error_type user_sim_invalid on its PromptResponse, so verifiers skip it as they skip a user-sim error. A built-in user sim's rewrite replaces the held answer in the reply count, so a done rewritten into a reply counts, and max_replies ends the conversation after it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…i/usersim-gate # Conflicts: # src/agent_env/task_step/task_steps/prompt_agent.py
…es its judge Through an endpoint, a Claude judge now drops the base's trailing /v1 (its client appends /v1/messages), and a bare name litellm can't place goes as openai/<name>. Before, both failed every call, and the gate fell back to its patterns alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…i/usersim-gate # Conflicts: # src/agent_env/task_step/task_steps/always_on.py
…d judge calls read_cap_seconds allowed READ_ASKS user-sim turns, and the gate's rewrites ran against the same deadline, so a slow sim could run out of time before max_rewrites and end an otherwise working run as user_sim_error. The cap now adds a user-sim turn per rewrite and, with a judge, a judge call per message checked; wake_reserve_seconds grows with it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cikowski
left a comment
There was a problem hiding this comment.
I like the shape of this: off by default, one run_gate shared by both loops, every verdict recorded, and a cut-off run marked so verifiers skip it. The flow tests are thorough. My concerns are with what the gate decides and with its inputs, not the plumbing.
-
The any-length phrases fail normal user messages. "I don't have access to that folder", "Should I send you the spreadsheet?", "The invoice has been sent to me already", "I'll pull the numbers from my bank tonight", "I apologize for the confusion" all fail
pattern_reasonstoday. A judge can't clear a pattern hit, so addingjudge_modeldoesn't help, and with the defaulton_fail: failthree of these in a row fail the step. I'd let the judge'sin_role: trueclear a phrase-only hit and tighten those phrases. -
Settings aren't type-checked.
{"max_rewrites": 1.5}and{"judge_min_chars": null}are accepted and raiseTypeErrormid-conversation;{"patterns": "false"}turns patterns on. -
messagein the read record is the held-back answer inprompt_agentbut the delivered rewrite inalways_on.user_sim_judgereads both, so it grades different things. Please pick one. -
If the judge can't be set up and
patternsis false, every message passes for the whole run with only a log line. A setup failure is deterministic, so I'd fail the step there and keep the fallback for failed calls.
Smaller: rate_user_satisfaction only special-cases user_sim_error, so a user_sim_invalid run is reported as "the agent's turn failed".
Heads up on ordering: this conflicts with #828 in _UserSide.ask and _ask_peer. I'd land #828 first. On the rebase, the hook's note should go on the first send only (not the rewrite request), and the judge should see it, since a user sim told what it found will say "I checked ... " and that is what this gate flags.
main sends every core model call through config/model_call.py, and no other core module may import LiteLLM. The user-sim gate's judge now resolves its route with model_call.resolve_route (judge_call_config) and calls model_call.complete, so its routing and cost record are core's; its own routing fix and the tests of it are dropped, with one test that the judge goes through that path. always_on keeps main's prescribe_formats import beside the gate's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lears phrase-only hits, typed settings - Phrases a user says no longer count: "I don't have access", "should I send", "I'll pull", "I can send it", "I apologize for the", "send me the results"; "has been sent/updated" counts only at a report's length. - With a judge, an in-role verdict clears a hit on a phrase alone (PHRASE_REASONS); a report's shape and length stays a failure. - UserSimGate checks each setting's type and range when the task loads, not mid-run. - prompt_agent's read holds the answer delivered, with first_message for the first one, as always_on's does; a message held back for good is a no_reply. - A judge that can't be set up fails the step when patterns are off, in both loops; with patterns on, they still stand alone. - rate_user_satisfaction skips a run the gate cut off as the user sim's doing, not the agent's. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
history[-0:] is the whole list, so 0 sent the entire conversation instead of none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#828 (a post-turn hook's note to the user sim) landed on main and met the gate in _UserSide. Both are kept, as the review asked: the note goes with the user sim's first message only, never with the gate's rewrite request, for a peer user sim (_ask_peer sends the first message with it through _send_to_peer, the rewrites without) and for the built-in one. The gate's judge is shown the note (run_gate and check take user_note; the judge's prompt says what the user found checking the work themselves), so a user sim reporting that check isn't read as the assistant. The note is recorded in user_side_notes on both paths, as #828 does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e-failed run isn't rated - A rewrite is parsed with the user-sim plugin's own parse(raw), as its answers are, so a plugin with another answer format keeps its rewrites instead of reading them as an empty no_reply. A plugin without parse can't support rewrites, so user_sim_gate with one is refused when the run starts. - rate_user_satisfaction skips an always_on run the gate failed as an error (on_fail "fail", error_type user_sim_invalid); a plain on_fail "end" sets no error and is still rated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
What
prompt_agentandalways_onget an opt-inuser_sim_gate. When the user side is an LLM user-sim, each of its messages ischecked before the agent receives it. That covers a peer user-sim and the built-in one
deploy_user_sim_agentdeploys:patterns(free): phrases and shapes that belong to the assistant's side, such as a report of workdone, a formatted deliverable, offers to do the work, or service phrases.
judge_model(optional): one model call over the last turns and the candidate. It decides only whosevoice the message is in. On its own it fails only messages of
judge_min_charsor more; shorter onesalso need a pattern.
A message that fails isn't sent. The user-sim is told why and asked to rewrite, up to
max_rewritestimes. If it still fails,
on_faildecides what happens:fail(the default) ends the conversation and fails the step, after thePromptResponseis recorded;endends the conversation only;sendrecords the verdicts and sends the message anyway.Every verdict is recorded in
context.metadata["user_sim_gate"][step_id], and an ended run is recorded incontext.metadata["user_sim_invalid"].The built-in user sim answers in JSON (
outcome,message,used), so its rewrite is parsed as its own answer:donewith text ends the conversation after the agent answers it, as before;doneends the conversation, as it would have.The read's record (
user_sim_reads) keeps what the user sim first answered, for anything grading the user sim, and lists each rewrite undergate_rewrites.In
always_oneach answer the user sim would send is checked at its read, before it's held for its response time:user_sim_invalid. Withon_fail: failit also setserror_type, so verifiers skip the run, as for a user-sim error.Her last read's answer, which never reaches the agent, isn't checked.
A step type that subclasses
PromptAgentTaskStepbut runs its own conversation would otherwise acceptuser_sim_gateand ignore it. Instead it refuses the setting at construction, unless it applies the gate and setsUSER_SIM_GATE_SUPPORTED = True, asalways_onnow does. For that, the check-and-rewrite loop isuser_sim_gate.run_gate(gate, text, history, rewrite):rewrite(note)returns the user-sim's new message, or None if the user side ended.prompt_agentuses the same function.{"user_sim_gate": {"judge_model": "<model>", "max_rewrites": 2, "on_fail": "fail"}}Why
An LLM playing the user can slip into the assistant's role. It answers its own question, reports work as
done, or offers to do the agent's job. After that, the rest of the run no longer tests the agent. Checking
each message, with a rewrite, keeps those messages out of the conversation, whether or not the user-sim
cooperates.
How it was tested
tst/unit/task_step/test_user_sim_gate.py: the patterns, judge parsing, the rendered history and therewrite note.
tst/unit/task_step/test_prompt_agent_user_side.py: 8 new tests over the real_UserSiderequest flow:failandend;run_gateon its own (rewrite, give up,send, stop);tst/unit/task_step/test_prompt_agent_built_in_user_sim.py: the built-in user sim held back and rewritten, failingafter its rewrites, and a rewrite that says nothing.
tst/unit/task_step/test_always_on_user_sim_gate.py: rewrite and deliver, fail (an error end),end, a pass withno rewrite, and no gate.
disagreements, checked against known cases):
with the next scripted request.
snapshot).
replay cover it.
always_on: two runs of a multi-day task, 4 wakes each, on the gate'spatterns plus a judge. One used the persona as written; the other's persona was told to answer like a status report.
only in the unit tests.
ruff checkis clean.No screenshots: the change has no UI.
Off by default: a step without
user_sim_gatebehaves exactly as before, andto_dictleaves the field out.🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge; both numbered previous findings are fixed.Summary
Adds an opt-in
user_sim_gatetoprompt_agentandalways_on. It checks simulated-user messages, requests rewrites, and records the results before delivery.Diagram
%%{init: {'theme': 'neutral'}}%% flowchart TD A[Simulated-user message] --> B[Check message] B -->|Pass| C[Deliver to agent] B -->|Fail, rewrites remain| D[Ask user sim to rewrite] D --> B B -->|Fail, no rewrites remain| E{on_fail} E -->|send| C E -->|end| F[End conversation] E -->|fail| G[End and record error]Reviews (16) · Last reviewed commit: "fix(always_on): a gate rewrite is read w..." · Reviewed by Greptile