Skip to content

feat(prompt_agent): let a post-turn hook give the user sim a note - #828

Merged
kaihackney-scale merged 3 commits into
mainfrom
kai/post-turn-hook-user-note
Nov 21, 2026
Merged

kaihackney-scale merged 3 commits into
mainfrom
kai/post-turn-hook-user-note

Conversation

@kaihackney-scale

@kaihackney-scale kaihackney-scale commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

A post-turn hook can already read the envs after each agent turn and decide what happens next, but it can't tell the
LLM user sim what it found. A hook that checks the agent's work (say, a sheet the agent claims to have made) needs the
user sim to react as the person would after looking: "I don't see the sheet yet". Without a channel, the sim judges
the work from the agent's own description of it.

turn.ask_user(note=...) adds that channel. The note goes to the user sim with the agent's reply, as something only
the sim sees: what the person found when checking the work themselves. The agent never sees it.

  • A peer user sim (a deployed agent named as the step's user) gets the note as one more text part after the reply.
    It is recorded under metadata["user_side_notes"].
  • The built-in user sim (deploy_user_sim_agent) gets it in its read as user_note, rendered as its own section. It
    is kept in the stored read.
  • A human peer gets no note.
  • ask_user() with no note behaves exactly as before. An empty or non-text note is refused.

How it was tested

  • New unit tests in test_prompt_agent_post_turn_hooks.py: the note reaches a peer user sim after the reply, is
    recorded, never reaches the agent, a human peer gets none, and a bad note is refused. In
    test_prompt_agent_built_in_user_sim.py: the built-in user sim's read carries it and the prompt shows it.
  • pytest tst/unit/task_step -k "prompt_agent or user_sim or post_turn": 820 passed. make lint: passed.
  • End to end: a local agent-env task run of a multi-turn task on deployed sandboxes, with a peer LLM user sim and a
    post-turn hook that checked the agent's work in the env after each turn and passed a note when something was
    missing. The notes reached the user sim, it raised them in its own words in its next message, the agent then
    fixed the missing item, and no note text appeared in anything the agent received.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no actionable issues were found.

Summary

Adds turn.ask_user(note=...) so a post-turn hook can tell the user sim what it found.

  • Post-turn hooks can give user sims a private work-check note.

No new actionable issues were found.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A["Post-turn hook"] --> B["turn.ask_user(note=...)"]
  B --> C{"Who answers?"}
  C -->|Trigger| D["Return trigger answer"]
  C -->|Human| E["Send agent reply only"]
  C -->|Peer sim| F["Send reply plus note"]
  C -->|Built-in sim| G["Add user_note to read"]
  F --> H["Record note"]
  G --> H
  F --> I["Return sim answer"]
  G --> I
  I --> J["Send answer to solver"]
Loading

Reviews (2) · Last reviewed commit: "Merge remote-tracking branch 'framework/..." · Reviewed by Greptile

turn.ask_user(note=...) shows the note to an LLM user sim with the agent's reply,
as what the person found checking the work themselves. The agent never sees it.
A peer user sim gets it as one more text part after the reply; a built-in user
sim gets it in its read as user_note. A human peer gets none. The note is
recorded under metadata user_side_notes (peer) or in the stored read (built-in).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kaihackney-scale
kaihackney-scale requested a review from a team as a code owner November 19, 2026 22:17

@cikowski cikowski left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. The note reaches both kinds of user sim, never the agent, and the no-note path is unchanged. Tests cover the wiring through the step for each path, which is what I wanted to see.

A few small things inline, none blocking.

One thing to plan for: this conflicts with #597 in _UserSide.ask and _ask_peer. I'd land this first and rebase #597 on it. When you do, the note should go on the first send to the peer only (not on the gate's rewrite request), and the gate's judge needs to see the note, otherwise a user sim that reports what it found gets flagged as writing in the assistant's voice.

kaihackney-scale and others added 2 commits November 20, 2026 18:59
…in sim's notes are recorded too

- The user side is asked once a turn: a note passed after it was already asked, without that note, now raises
  instead of being dropped with no trace. One passed after an agent trigger answered the turn has no one to reach,
  which is documented, not an error.
- A note to a user sim deploy_user_sim_agent deployed is recorded in metadata["user_side_notes"] too, as the peer
  path's is, so there is one place to look for what the user sim was told.
- ask's docstring says a note reaches no one when an agent trigger answers the turn or once end_after is set.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kaihackney-scale
kaihackney-scale merged commit 1cf6421 into main Nov 21, 2026
35 checks passed
@kaihackney-scale
kaihackney-scale deleted the kai/post-turn-hook-user-note branch November 21, 2026 00:15
kaihackney-scale added a commit that referenced this pull request Nov 21, 2026
#828 (a post-turn hook's note to the user sim) landed on main and met the gate in _UserSide. Both are kept, as the
review asked: the note goes with the user sim's first message only, never with the gate's rewrite request, for a peer
user sim (_ask_peer sends the first message with it through _send_to_peer, the rewrites without) and for the built-in
one. The gate's judge is shown the note (run_gate and check take user_note; the judge's prompt says what the user found
checking the work themselves), so a user sim reporting that check isn't read as the assistant. The note is recorded
in user_side_notes on both paths, as #828 does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants