Skip to content

fix(agents): Hermes, Pydantic AI, pi, OpenHands and browser-use count every model call a turn makes - #855

Open
earakely-scale wants to merge 3 commits into
mainfrom
edgararakelyan/agents-count-every-call
Open

earakely-scale wants to merge 3 commits into
mainfrom
edgararakelyan/agents-count-every-call

Conversation

@earakely-scale

@earakely-scale earakely-scale commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

What changes

P8's usage audit (FD-5980, #850) ran 80 scenarios across every shipped agent against Scale's LiteLLM proxy, with the proxy's own usage for every call beside each run's stored entries. Five agents that already report usage missed calls or priced them wrong. Each now counts every model call its turn made, or marks the ones it can't count in unreported_calls (FD-6449).

  • Hermes (hermes/):
    • Its auxiliary calls (title generation, compression) are added from the usage file's auxiliary block. Hermes counts them canonically (input without the cache), so the cache is added back.
    • A delegate_task child's calls, which the usage file leaves out, are counted from their post_api_request events. The plugin now keeps each event's usage. The bridge tells the child's events from the turn's own by turn_id, and the session stays the parent's.
    • A turn killed before it wrote its usage file counts the calls it made from their events, plus one more as unreported: the call in flight, or its title call.
    • The background review is turned off. It replays the turn on the main model in a thread the turn's exit kills before it reports its calls, which the proxy bills anyway; Hermes turns it off for its own one-shot turns too.
    • Each part (main calls, auxiliary calls, a child's calls) is counted, or marked unreported, on its own, so a main answer without usage keeps the others' counts.
    • A cost is kept only when Hermes priced every call the turn made (cost_status estimated or actual, no child, nothing unreported). For a model it can't price it says unknown and $0, which was stored as $0.
  • Pydantic AI (pydantic-ai/): a subagent's calls (an agent run from a tool) are counted, through a Hooks(after_model_request=...) capability that sees every model response in the process's turn.
  • pi (pi/):
    • A compaction's summary call is counted from compaction_end, the only place pi reports it.
    • A request the endpoint refused (pi ends it as an error message with no usage) counts nothing. Before, it counted as an unreported call, marking a complete turn partial.
  • OpenHands (openhands/): on Claude, the cache was added to an input that already held it, so the cost came out 2.3 to 3.7 times OpenHands' own figure. A call's input is now taken as holding the cache when its prompt is at least its cache reads and writes.
  • browser-use (browser-use/):
    • A request cut off mid-call is counted as unreported, from the SDK's own HTTP hooks (sent, refused, answered). It was stored as $0, unmarked.
    • The tool-call count is the turn's own; it added up across a context's turns before.
  • src/agent_env/agents/AGENTS.md: a cost the runtime couldn't price is unset, never 0.

Decisions to review

  • OpenHands reads a Chat Completions prompt at least as large as its cache as counting it. LiteLLM hands OpenHands every call in OpenAI's shape, which can't say whether the prompt holds the cache. Before, such a call left the flag unknown, and P10 priced the cache as extra input, a marked upper bound: 2.3 to 3.7 times OpenHands' own figure on Claude, whose prompt through Scale's proxy now counts the cache. OpenAI's rule makes those exact. The cost: an endpoint whose prompt leaves the cache out (as Scale's proxy sent Claude's before smig#817), on a call whose uncached part is larger than its cache, would now be short, unmarked. A prompt smaller than its cache still gets the cache added. The integration test's GPT call, whose prompt counts its cache, now reads as counting it instead of unknown.

Tests

  • Hermes, on recorded events: auxiliary calls; a priced and an unpriced cost; a delegate_task child (45,484 in, 172 out, the proxy's figures, with the session kept the parent's); a killed turn (26,229 in, 134 out, one unreported); and the review fork off in the config.
  • Pydantic AI: a subagent's calls are counted.
  • pi: a compaction's summary call (the recorded pi-compaction-gpt turn equals the proxy's three calls), one that ended without usage, and a refused request.
  • OpenHands: the recorded Claude turn is priced at its own figure ($0.01595985).
  • browser-use: a cut-off request, a mix of answered and cut-off calls, a refused request, and a follow-up turn's tool count.
  • Hermes, from Greptile's review: a priced turn with a delegate_task child keeps no cost of its own, since Hermes' figure leaves out the child's calls; and when the main calls report no counts, the title call's and the child's are kept and only the main calls are unreported.
  • Pydantic AI on its pinned runtime: the unit tier can't import Pydantic AI (it's only in the agent's image), so a script ran this branch's _agent on pydantic-ai-slim 2.61.0 and pydantic-ai-harness 0.61.0 with a scripted model. The parent delegated to itself, the child answered, the parent answered: 3 calls, each recorded and summed once.
  • Each new test fails with its fix taken out. All 1,123 agent tests pass.

E2E against the real proxy

On an admin host, on this branch's images, the audit's 36 scenarios for these five agents, through a logging pass-through in front of Scale's LiteLLM proxy. Each run's stored entries were compared with the proxy's usage for every call that role's model made.

Agent Scenarios Result Was
Hermes baseline (Claude, GPT), chat, delegate_task, review fork, 40% injected 500s, chat greeting, killed turn exact; the killed turn a marked lower bound aux, child and killed calls missing; $0 for unpriced
Pydantic AI baseline (Claude, GPT), chat, subagent (Claude, GPT), cache probe, 40% injected 500s exact subagent calls missing
pi baseline (Claude, GPT), chat, compaction, cache probe, host timeout, 40% injected 500s exact compaction's call missing
OpenHands baseline (Claude, GPT), chat (Claude, GPT), condenser (44 calls), host timeout, 40% injected 500s exact, and the cost equals OpenHands' own on Claude Claude's cost 2.3 to 3.7 times too high
browser-use baseline (Claude, GPT), chat (Claude, GPT), long cached prompt, Sonnet cache, cut-off request exact; the cut-off request marked unreported the cut-off request $0, unmarked

43 of 45 role entries were exact, user simulators included; the other two were the killed Hermes turn and the cut-off browser-use request, each correctly marked. The rerun also showed pi marking a fault run partial for a refused request, which the last fix here addresses.

Chaos

88 random runs over 27 minutes on two workers (the first minutes building the five images), across 19 of the audit's scenarios for these agents, through the same pass-through. 45 had 20% or 35% of their calls answered with an injected 500, so the agents retried: 446 calls were answered and 100 refused.

Every role entry, 88 solvers and 28 user simulators, equals the proxy's usage for its answered calls, and is priced. The one partial entry was a killed Hermes turn: its counts are exact, and its one call that may have been in flight is marked.

The chaos ran on ecb5ac2. Greptile's round then changed Hermes (fb4fce1), so its six scenarios ran again on the head: a delegate_task child, a chat, the review fork, a killed turn, and two with injected 500s. All six are exact, the killed turn marked.

The e2e, the chaos and the rerun came to about $2.61 of model calls at list price.

Images

Each of the five agents' folders changed, so each gets a new image at the next release, promoted on its own (S6).

Docs

For FD-6263 (P7/P8 items) and FD-6442: Hermes, Pydantic AI, pi, OpenHands and browser-use count every call a turn makes, subagents', side calls and compactions included; a refused request counts nothing; a cut-off or killed call is unreported; an agent's unpriced cost is unset, never $0.

🤖 Generated with Claude Code

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no new actionable issues remain.

Summary

Counts previously missed model calls across five agents and leaves unpriced Hermes costs unset.

  • Hermes usage now includes auxiliary, delegated, and interrupted-turn calls.
  • Pydantic AI usage now includes model calls from delegated runs.
  • pi usage now includes compaction calls and skips refused requests.
  • OpenHands counts cached input once when pricing model calls.
  • browser-use marks unanswered calls and counts only the current turn's tools.

Reviews (3) · Last reviewed commit: "test(agents): OpenHands reads a Chat Com..." · Reviewed by Greptile

… every model call a turn makes

Hermes counts its auxiliary calls, its delegate_task children's calls and a killed turn's calls, and
leaves a cost it couldn't price unset; Pydantic AI counts its subagents' calls; pi counts its
compactions' summary calls and counts a request its endpoint refused as nothing; OpenHands stops
adding Claude's cache to an input that already holds it; browser-use marks a request cut off
mid-call as unreported and counts a turn's tool calls only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@earakely-scale
earakely-scale requested a review from a team as a code owner November 21, 2026 00:10
earakely-scale and others added 2 commits November 20, 2026 16:16
… that reported none, and no cost that misses a child's calls

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…arge as its cache as counting it

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant