Repository navigation
fix(agents): Hermes, Pydantic AI, pi, OpenHands and browser-use count every model call a turn makes - #855
Open
earakely-scale wants to merge 3 commits into
Open
fix(agents): Hermes, Pydantic AI, pi, OpenHands and browser-use count every model call a turn makes#855earakely-scale wants to merge 3 commits into
earakely-scale wants to merge 3 commits into
Conversation
… every model call a turn makes Hermes counts its auxiliary calls, its delegate_task children's calls and a killed turn's calls, and leaves a cost it couldn't price unset; Pydantic AI counts its subagents' calls; pi counts its compactions' summary calls and counts a request its endpoint refused as nothing; OpenHands stops adding Claude's cache to an input that already holds it; browser-use marks a request cut off mid-call as unreported and counts a turn's tool calls only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… that reported none, and no cost that misses a child's calls Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…arge as its cache as counting it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
P8's usage audit (FD-5980, #850) ran 80 scenarios across every shipped agent against Scale's LiteLLM proxy, with the proxy's own usage for every call beside each run's stored entries. Five agents that already report usage missed calls or priced them wrong. Each now counts every model call its turn made, or marks the ones it can't count in
unreported_calls(FD-6449).hermes/):auxiliaryblock. Hermes counts them canonically (input without the cache), so the cache is added back.delegate_taskchild's calls, which the usage file leaves out, are counted from theirpost_api_requestevents. The plugin now keeps each event'susage. The bridge tells the child's events from the turn's own byturn_id, and the session stays the parent's.cost_statusestimatedoractual, no child, nothing unreported). For a model it can't price it saysunknownand $0, which was stored as $0.pydantic-ai/): a subagent's calls (an agent run from a tool) are counted, through aHooks(after_model_request=...)capability that sees every model response in the process's turn.pi/):compaction_end, the only place pi reports it.openhands/): on Claude, the cache was added to an input that already held it, so the cost came out 2.3 to 3.7 times OpenHands' own figure. A call's input is now taken as holding the cache when its prompt is at least its cache reads and writes.browser-use/):src/agent_env/agents/AGENTS.md: a cost the runtime couldn't price is unset, never 0.Decisions to review
Tests
delegate_taskchild (45,484 in, 172 out, the proxy's figures, with the session kept the parent's); a killed turn (26,229 in, 134 out, one unreported); and the review fork off in the config.delegate_taskchild keeps no cost of its own, since Hermes' figure leaves out the child's calls; and when the main calls report no counts, the title call's and the child's are kept and only the main calls are unreported._agenton pydantic-ai-slim 2.61.0 and pydantic-ai-harness 0.61.0 with a scripted model. The parent delegated to itself, the child answered, the parent answered: 3 calls, each recorded and summed once.E2E against the real proxy
On an admin host, on this branch's images, the audit's 36 scenarios for these five agents, through a logging pass-through in front of Scale's LiteLLM proxy. Each run's stored entries were compared with the proxy's usage for every call that role's model made.
delegate_task, review fork, 40% injected 500s, chat greeting, killed turn43 of 45 role entries were exact, user simulators included; the other two were the killed Hermes turn and the cut-off browser-use request, each correctly marked. The rerun also showed pi marking a fault run partial for a refused request, which the last fix here addresses.
Chaos
88 random runs over 27 minutes on two workers (the first minutes building the five images), across 19 of the audit's scenarios for these agents, through the same pass-through. 45 had 20% or 35% of their calls answered with an injected 500, so the agents retried: 446 calls were answered and 100 refused.
Every role entry, 88 solvers and 28 user simulators, equals the proxy's usage for its answered calls, and is priced. The one partial entry was a killed Hermes turn: its counts are exact, and its one call that may have been in flight is marked.
The chaos ran on ecb5ac2. Greptile's round then changed Hermes (fb4fce1), so its six scenarios ran again on the head: a
delegate_taskchild, a chat, the review fork, a killed turn, and two with injected 500s. All six are exact, the killed turn marked.The e2e, the chaos and the rerun came to about $2.61 of model calls at list price.
Images
Each of the five agents' folders changed, so each gets a new image at the next release, promoted on its own (S6).
Docs
For FD-6263 (P7/P8 items) and FD-6442: Hermes, Pydantic AI, pi, OpenHands and browser-use count every call a turn makes, subagents', side calls and compactions included; a refused request counts nothing; a cut-off or killed call is unreported; an agent's unpriced cost is unset, never $0.
🤖 Generated with Claude Code


Confidence Score: 5/5
The PR appears safe to merge; no new actionable issues remain.Summary
Counts previously missed model calls across five agents and leaves unpriced Hermes costs unset.
Reviews (3) · Last reviewed commit: "test(agents): OpenHands reads a Chat Com..." · Reviewed by Greptile