Skip to content

feat(prompt_agent): an opt-in gate on user-sim messages - #597

Open
kaihackney-scale wants to merge 18 commits into
mainfrom
kai/usersim-gate
Open

kaihackney-scale wants to merge 18 commits into
mainfrom
kai/usersim-gate

Conversation

@kaihackney-scale

@kaihackney-scale kaihackney-scale commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator

What

prompt_agent and always_on get an opt-in user_sim_gate. When the user side is an LLM user-sim, each of its messages is
checked before the agent receives it. That covers a peer user-sim and the built-in one deploy_user_sim_agent deploys:

  • patterns (free): phrases and shapes that belong to the assistant's side, such as a report of work
    done, a formatted deliverable, offers to do the work, or service phrases.
  • judge_model (optional): one model call over the last turns and the candidate. It decides only whose
    voice the message is in. On its own it fails only messages of judge_min_chars or more; shorter ones
    also need a pattern.

A message that fails isn't sent. The user-sim is told why and asked to rewrite, up to max_rewrites
times. If it still fails, on_fail decides what happens:

  • fail (the default) ends the conversation and fails the step, after the PromptResponse is recorded;
  • end ends the conversation only;
  • send records the verdicts and sends the message anyway.

Every verdict is recorded in context.metadata["user_sim_gate"][step_id], and an ended run is recorded in
context.metadata["user_sim_invalid"].

The built-in user sim answers in JSON (outcome, message, used), so its rewrite is parsed as its own answer:

  • a reply goes on to the gate's next check;
  • a done with text ends the conversation after the agent answers it, as before;
  • a no_reply or an empty done ends the conversation, as it would have.

The read's record (user_sim_reads) keeps what the user sim first answered, for anything grading the user sim, and lists each rewrite under gate_rewrites.

In always_on each answer the user sim would send is checked at its read, before it's held for its response time:

  • a rewrite is asked for in the user sim's own session;
  • the read's timeline event records the verdicts, the rewrites and the first message;
  • an answer that still fails is never delivered, and the run ends with user_sim_invalid. With on_fail: fail it also sets error_type, so verifiers skip the run, as for a user-sim error.

Her last read's answer, which never reaches the agent, isn't checked.

A step type that subclasses PromptAgentTaskStep but runs its own conversation would otherwise accept user_sim_gate and ignore it. Instead it refuses the setting at construction, unless it applies the gate and sets USER_SIM_GATE_SUPPORTED = True, as always_on now does. For that, the check-and-rewrite loop is user_sim_gate.run_gate(gate, text, history, rewrite): rewrite(note) returns the user-sim's new message, or None if the user side ended. prompt_agent uses the same function.

{"user_sim_gate": {"judge_model": "<model>", "max_rewrites": 2, "on_fail": "fail"}}

Why

An LLM playing the user can slip into the assistant's role. It answers its own question, reports work as
done, or offers to do the agent's job. After that, the rest of the run no longer tests the agent. Checking
each message, with a rewrite, keeps those messages out of the conversation, whether or not the user-sim
cooperates.

How it was tested

  • Unit tests:
    • tst/unit/task_step/test_user_sim_gate.py: the patterns, judge parsing, the rendered history and the
      rewrite note.
    • tst/unit/task_step/test_prompt_agent_user_side.py: 8 new tests over the real _UserSide request flow:
      • pass, rewrite and recover;
      • fail after the rewrites, both fail and end;
      • the judge failing a message the patterns pass;
      • a judge error falling back to the patterns;
      • a short message not failed by the judge alone;
      • dict round-trip and validation;
      • run_gate on its own (rewrite, give up, send, stop);
      • a subclass with its own loop refusing the gate unless it supports it.
    • tst/unit/task_step/test_prompt_agent_built_in_user_sim.py: the built-in user sim held back and rewritten, failing
      after its rewrites, and a rewrite that says nothing.
    • tst/unit/task_step/test_always_on_user_sim_gate.py: rewrite and deliver, fail (an error end), end, a pass with
      no rewrite, and no gate.
  • Offline, against labels of real user-sim messages (two independent model labelers, with an adjudicator for
    disagreements, checked against known cases):
    • every labeled out-of-role message is caught;
    • under 0.5% of in-role messages are failed;
    • on held-out messages the patterns alone flag 0.22% of in-role messages.
  • Rewrite recovery: at real out-of-role points, asked to rewrite, the user-sim recovered every time,
    with the next scripted request.
  • E2E: gated and ungated runs of the same multi-turn tasks, through the full task (deploy, conversation,
    snapshot).
    • Every run completed.
    • The gate recorded a verdict for every user-sim message.
    • No out-of-role message reached the agent, judged separately by a grader with the scenario.
    • The rewrite path was not triggered live, because nothing failed; the unit tests and the offline
      replay cover it.
  • E2E on the built-in user sim in always_on: two runs of a multi-day task, 4 wakes each, on the gate's
    patterns plus a judge. One used the persona as written; the other's persona was told to answer like a status report.
    • Both completed.
    • Every reply the agent would get was checked, and passed.
    • The persona change didn't make the user sim write as the assistant, so the rewrite and fail paths again ran
      only in the unit tests.
    • Every sandbox was torn down.

ruff check is clean.

No screenshots: the change has no UI.

Off by default: a step without user_sim_gate behaves exactly as before, and to_dict leaves the field out.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; both numbered previous findings are fixed.

Summary

Adds an opt-in user_sim_gate to prompt_agent and always_on. It checks simulated-user messages, requests rewrites, and records the results before delivery.

  • Prompt-agent conversations check simulated-user replies before the assistant receives them.
  • Always-on runs check each eligible simulated-user answer before delivering it.
  • Satisfaction grading treats gate cutoffs as user-sim outcomes, not agent failures.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Simulated-user message] --> B[Check message]
  B -->|Pass| C[Deliver to agent]
  B -->|Fail, rewrites remain| D[Ask user sim to rewrite]
  D --> B
  B -->|Fail, no rewrites remain| E{on_fail}
  E -->|send| C
  E -->|end| F[End conversation]
  E -->|fail| G[End and record error]
Loading

Reviews (16) · Last reviewed commit: "fix(always_on): a gate rewrite is read w..." · Reviewed by Greptile

prompt_agent's new user_sim_gate checks each LLM user-sim message before the agent receives it: phrase
patterns for the assistant's side of a conversation, and optionally a small judge model that decides only
whose voice the message is in. A message that fails is held back and the user-sim is asked to rewrite it,
up to max_rewrites times; on_fail then fails the step, ends the conversation, or sends it anyway. Every
verdict is recorded in context.metadata["user_sim_gate"].

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e its cutoffs

`user_sim_gate={}` now enables the gate with its defaults; only None leaves it off. The patterns' length and
count cutoffs, and how much of a judge's reason and error a verdict keeps, are named class constants.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale

Copy link
Copy Markdown
Collaborator

🤖 Automated Claude routine — Env Pod PR Context Review. This is a bot, not Edgar Arakelyan; not a human review.

Context from outside this diff, in case it's useful: there's a second open PR adding a step type that subclasses PromptAgentTaskStep and filters its inherited fields by an explicit list, so user_sim_gate would land on it as a silently-inert field.

  • This PR: src/agent_env/task_step/task_steps/prompt_agent.py:181 adds user_sim_gate as a constructor kwarg on PromptAgentTaskStep, serialized at :309-310 / :348, and enforced inside _UserSide (:1063, :1130).
  • feat(always_on): a step that runs an assistant over days of task time, with a simulated user (FD-5364) #566 (feat(always_on): a step that runs an assistant over days of task time, with a simulated user, head 974a6aba): src/agent_env/task_step/task_steps/always_on.py:201 is class AlwaysOnTaskStep(PromptAgentTaskStep). Its __init__ takes **prompt_fields and pops a fixed drop list at :217 (max_conversation_turns, user_agent_name, user_a2a_url, post_turn_hooks, user_output_format, user_model, snapshot_config), then forwards the rest to super().__init__. Its to_dict at :303-304 starts from TaskStep.to_dict(self) — not the parent's — and copies only _PROMPT_FIELDS (:252) plus _OWN.

So in either merge order, an always_on step authored with user_sim_gate is accepted at construction (it isn't in the drop list), dropped on to_dict (it isn't in _PROMPT_FIELDS), and never consulted: always_on.py has its own _Run loop and contains no reference to _UserSide, _converse or user_sim_gate. The :588 warning only fires on the prompt_agent path, so there'd be no signal either. Deciding where it goes — a third list entry, the drop list, or out of scope — is probably cheaper now than after both land.

Related, same direction: #567 addresses the same failure mode from the prompt side — user_sim_versions.py builds a system prompt around "You are not the AI assistant … Don't do your assistant's work: never look things up for it or hand over facts it should find", and user_sim.md reports a measured effect (done collapsed into the same message 9/10 → 1/10). Worth a look if you want the gate's pattern list and that prompt to agree on what counts as the assistant's voice.


Generated by Claude Code

t and others added 2 commits November 10, 2026 13:52
…te where it would be ignored

The check-and-rewrite loop moves to user_sim_gate.run_gate, with a rewrite callback, so a step type that runs
its own conversation can apply the gate; prompt_agent's user side now calls it. A PromptAgentTaskStep subclass
that replaces prompt_agent's conversation refuses user_sim_gate at construction, instead of accepting and
ignoring it, unless it sets USER_SIM_GATE_SUPPORTED = True.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main reworked prompt_agent's user side (the built-in user sim, end_after). The gate keeps its
history wrapper in ask(), main's end_after short-circuit opens _answer(), and the gate still
checks peer user-sim messages. A gate set with the built-in user sim is warned about, as with a
non-user-sim peer, since those messages aren't checked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
t and others added 4 commits November 12, 2026 17:18
…p errors fall back to the patterns

- AlwaysOnTaskStep.from_dict passes user_sim_gate on, so a saved step is refused as one built in
  code is, rather than running with the setting dropped.
- A judge whose setup fails (its key or endpoint) no longer ends the conversation: like a failed
  judge call, the patterns' verdict stands and the error is recorded on each verdict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ps checking

The fallback rebuilt the gate without its judge, which a judge-only gate (patterns off) refuses,
so its setup error ended the conversation. The gate's check now takes judge_unavailable instead:
the judge isn't called, the patterns' verdict stands (a pass with patterns off), and the reason is
recorded on the verdict, as for a judge call that fails. run_gate passes it only when set, so a
check override with the old signature still works.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…agent and always_on

prompt_agent applies user_sim_gate to the user deploy_user_sim_agent deployed: a held-back answer is rewritten in its session and parsed as its own answer, and the read's record keeps the first answer and lists each rewrite. always_on applies it at each read before an answer is held: one that still fails is never delivered and ends the run (user_sim_invalid; an error with on_fail fail).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
t and others added 2 commits November 16, 2026 13:07
…ite counts toward max_replies

A conversation the gate cut off with on_fail fail now records error_type user_sim_invalid on its PromptResponse, so verifiers skip it as they skip a user-sim error. A built-in user sim's rewrite replaces the held answer in the reply count, so a done rewritten into a reply counts, and max_replies ends the conversation after it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…i/usersim-gate

# Conflicts:
#	src/agent_env/task_step/task_steps/prompt_agent.py
kaihackney-scale and others added 2 commits November 19, 2026 22:31
…es its judge

Through an endpoint, a Claude judge now drops the base's trailing /v1 (its client appends /v1/messages), and a bare
name litellm can't place goes as openai/<name>. Before, both failed every call, and the gate fell back to its
patterns alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…i/usersim-gate

# Conflicts:
#	src/agent_env/task_step/task_steps/always_on.py
…d judge calls

read_cap_seconds allowed READ_ASKS user-sim turns, and the gate's rewrites ran against the same deadline, so a slow
sim could run out of time before max_rewrites and end an otherwise working run as user_sim_error. The cap now adds a
user-sim turn per rewrite and, with a judge, a judge call per message checked; wake_reserve_seconds grows with it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@cikowski cikowski left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the shape of this: off by default, one run_gate shared by both loops, every verdict recorded, and a cut-off run marked so verifiers skip it. The flow tests are thorough. My concerns are with what the gate decides and with its inputs, not the plumbing.

  1. The any-length phrases fail normal user messages. "I don't have access to that folder", "Should I send you the spreadsheet?", "The invoice has been sent to me already", "I'll pull the numbers from my bank tonight", "I apologize for the confusion" all fail pattern_reasons today. A judge can't clear a pattern hit, so adding judge_model doesn't help, and with the default on_fail: fail three of these in a row fail the step. I'd let the judge's in_role: true clear a phrase-only hit and tighten those phrases.

  2. Settings aren't type-checked. {"max_rewrites": 1.5} and {"judge_min_chars": null} are accepted and raise TypeError mid-conversation; {"patterns": "false"} turns patterns on.

  3. message in the read record is the held-back answer in prompt_agent but the delivered rewrite in always_on. user_sim_judge reads both, so it grades different things. Please pick one.

  4. If the judge can't be set up and patterns is false, every message passes for the whole run with only a log line. A setup failure is deterministic, so I'd fail the step there and keep the fallback for failed calls.

Smaller: rate_user_satisfaction only special-cases user_sim_error, so a user_sim_invalid run is reported as "the agent's turn failed".

Heads up on ordering: this conflicts with #828 in _UserSide.ask and _ask_peer. I'd land #828 first. On the rebase, the hook's note should go on the first send only (not the rewrite request), and the judge should see it, since a user sim told what it found will say "I checked ... " and that is what this gate flags.

kaihackney-scale and others added 2 commits November 20, 2026 17:08
main sends every core model call through config/model_call.py, and no other core module may import LiteLLM. The
user-sim gate's judge now resolves its route with model_call.resolve_route (judge_call_config) and calls
model_call.complete, so its routing and cost record are core's; its own routing fix and the tests of it are dropped,
with one test that the judge goes through that path. always_on keeps main's prescribe_formats import beside the gate's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lears phrase-only hits, typed settings

- Phrases a user says no longer count: "I don't have access", "should I send", "I'll pull", "I can send it",
  "I apologize for the", "send me the results"; "has been sent/updated" counts only at a report's length.
- With a judge, an in-role verdict clears a hit on a phrase alone (PHRASE_REASONS); a report's shape and length stays
  a failure.
- UserSimGate checks each setting's type and range when the task loads, not mid-run.
- prompt_agent's read holds the answer delivered, with first_message for the first one, as always_on's does; a
  message held back for good is a no_reply.
- A judge that can't be set up fails the step when patterns are off, in both loops; with patterns on, they still
  stand alone.
- rate_user_satisfaction skips a run the gate cut off as the user sim's doing, not the agent's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
kaihackney-scale and others added 2 commits November 20, 2026 17:19
history[-0:] is the whole list, so 0 sent the entire conversation instead of none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#828 (a post-turn hook's note to the user sim) landed on main and met the gate in _UserSide. Both are kept, as the
review asked: the note goes with the user sim's first message only, never with the gate's rewrite request, for a peer
user sim (_ask_peer sends the first message with it through _send_to_peer, the rewrites without) and for the built-in
one. The gate's judge is shown the note (run_gate and check take user_note; the judge's prompt says what the user found
checking the work themselves), so a user sim reporting that check isn't read as the assistant. The note is recorded
in user_side_notes on both paths, as #828 does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e-failed run isn't rated

- A rewrite is parsed with the user-sim plugin's own parse(raw), as its answers are, so a plugin with another answer
  format keeps its rewrites instead of reading them as an empty no_reply. A plugin without parse can't support
  rewrites, so user_sim_gate with one is refused when the run starts.
- rate_user_satisfaction skips an always_on run the gate failed as an error (on_fail "fail", error_type
  user_sim_invalid); a plain on_fail "end" sets no error and is still rated.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants