Skip to content

Correct the August thinking note's account of litellm's reasoning count - #265

Merged
MaxGhenis merged 4 commits into
mainfrom
august-note-litellm-correction
Nov 15, 2026
Merged

MaxGhenis merged 4 commits into
mainfrom
august-note-litellm-correction

Conversation

@MaxGhenis

@MaxGhenis MaxGhenis commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

The August thinking note (sensitivity/claude-thinking-2026-08.md) said litellm "does not populate reasoning-token fields for Anthropic even when thinking blocks are present". Its Claude Haiku 5.5 section said "litellm reports no reasoning tokens for Anthropic", and its introduction said "every Claude row logged zero reasoning tokens". litellm fills the field from a count of the thinking text a response returns, and Anthropic can return a thinking block with empty text. Claude Fable 5's rows in the board's predictions file record no usage. In #261 we found how litellm counts and left the note unchanged.

What changes

  • The second finding now states how litellm counts. It gives a recorded Claude Haiku 5.5 probe as an example: Anthropic counted 119 of the response's 123 output tokens as thinking but returned the thinking block with empty text, and litellm counts 0 for it.
  • The second finding also states what Record Anthropic's thinking-token count as reasoning_tokens #261 changed. The harness records Anthropic's own count when a new response reports a whole number no larger than its output-token count, and leaves reasoning_tokens empty for any other reported count. For a response that reports no count, it still records litellm's count on the sync path and none on the batch path. When it replays a response from an older cache entry, it keeps litellm's count. The note tells readers to read a 0 in these runs' reasoning_tokens as not counted.
  • The introduction now says every Claude row with recorded usage logged zero reasoning tokens, and that Claude Fable 5's rows in the board's predictions file record no usage.
  • The line before the two findings now says PolicyBench verified only the first with live requests from the harness's request builder.
  • The Claude Haiku 5.5 section now says the harness did not yet read Anthropic's thinking-token count, so the run records 0 reasoning tokens.
  • A new section, "Correction (November 2026)", quotes the three earlier claims.
  • No score, rank or cost changes.

Evidence

  • litellm 1.88.1 and 1.95.0 set reasoning_tokens from a token count of the returned thinking text and never read Anthropic's thinking_tokens (llms/anthropic/chat/transformation.py). The lockfile pinned 1.88.1 for all five of the note's sensitivity runs and has pinned 1.95.0 since November 2.
  • The recorded Claude Haiku 5.5 response in tests/fixtures/anthropic_haiku55_probe_responses.json returns one thinking block with empty text, where Anthropic counts 119 of 123 output tokens as thinking. tests/test_anthropic_thinking_tokens.py replays it through litellm, which counts 0.
  • Each of the five runs records 0 reasoning tokens on all 1,984 rows. The committed predictions keep no thinking blocks, so the note says nothing about what those runs' responses returned.
  • In the board snapshot's predictions file, Claude Fable 5's 1,984 rows record no usage, and the other 11 Claude models record 0 on every row.
  • _bounded_thinking_tokens in policybench/eval_no_tools.py sets the bound on Anthropic's count, and docs/runbook.md section 8b lists the cases that still record litellm's count or none.

Test

test_doc_reads_the_runs_zero_reasoning_tokens_as_not_counted checks the five runs' counts in the committed predictions. In the board's predictions it checks Claude Fable 5's missing usage and the other 11 Claude models' 0 reasoning tokens. It also checks the probe's block and counts in the fixture. It requires the note's body to cite the probe and its two counts, to tell readers to read a 0 as "not counted", and to hold no sentence that matches a withdrawn claim's pattern. It requires the correction section, whatever its date, to quote each withdrawn claim in full and to say that no score, rank or cost changed. It fails on main's note.

Local: tests/test_sensitivity_evidence.py and tests/test_disclosures.py, 55 passed.

🤖 Generated with Claude Code

The note's second finding said litellm "does not populate reasoning-token
fields for Anthropic even when thinking blocks are present", and its Claude
Haiku 5.5 section said "litellm reports no reasoning tokens for Anthropic".
litellm fills the field from a count of the thinking text a response
returns. In the note's runs Claude returned its thinking blocks with empty
text, so litellm counted 0.

- Reword both passages and add a correction section that quotes the earlier
  sentences. No score, rank or cost changes.
- Add a test: each of the note's five runs records completion tokens and 0
  reasoning tokens on every row, Claude Fable 5's board rows record no
  usage, and only the correction section holds the two earlier sentences.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
policybench-site Ready Ready Preview Nov 15, 2026 1:44am UTC

Request Review

A review of the correction found four faults, all fixed here.

The note said the five runs' responses returned thinking blocks with empty
text. The committed predictions keep no thinking blocks, so nothing shows
what those responses returned. The note now states the runs' zeros and cites
the recorded Claude Haiku 5.5 probe as the example: Anthropic counted 119 of
123 output tokens as thinking, the thinking block came back empty, and
litellm counted 0.

The note said the harness records Anthropic's count "whenever a response
reports one". The harness records a count only when it is a whole number
from 0 to the response's output tokens, and a response replayed from an
older cache entry keeps litellm's count. The note now says both.

The introduction said every Claude row logged zero reasoning tokens. Claude
Fable 5's board rows record no usage. The introduction now limits the claim
to rows with recorded usage, and the correction quotes the earlier claim.

The test fixed whole sentences of prose and a dated heading. It now checks
the data (the runs' counts, Claude Fable 5's missing usage, the probe's
block and counts), that the note's body cites the probe and reads a 0 as
"not counted", that no withdrawn claim returns to the body, and that the
correction quotes each withdrawn claim in full under any date.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…e test's gaps

Two more readers checked the correction and found no false statement. This
commit takes their smaller points.

The note:
- names Claude Fable 5's missing usage in the introduction, beside the
  claim it limits, and says the rows are those in the board's predictions
  file;
- gives Anthropic the verb in the probe example and says litellm counts 0
  for the recorded response, which the thinking-token test replays;
- states what the harness does since #261 for a response with no count
  (litellm's count on the sync path, none on the batch path) and for a
  replay from an older cache entry, and leaves the bound to the runbook;
- splits the Claude Haiku 5.5 sentence in two;
- tightens the correction section and keeps its three quotations.

The test:
- searches the body for each withdrawn claim without regard to case and with
  the nearby wordings (the field or fields; no, zero or 0; Anthropic or
  Claude; logged or recorded);
- finds the correction section by its heading, whatever follows the word,
  and treats any section after it as body;
- requires the probe's two counts in order;
- documents the check that the correction says no score, rank or cost
  changed.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…icyBench verified live

The test now checks that the 11 Claude models other than Claude Fable 5
record completion tokens and 0 reasoning tokens on every row of the board's
predictions, as the note's introduction says. The note's lead-in to its two
findings says PolicyBench verified only the first with live requests. The
second finding states the bound on Anthropic's count and the cases where
the harness still records litellm's count or none.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@MaxGhenis

Copy link
Copy Markdown
Contributor Author

Review record at 0fd59e7:

  • Round 1 (Subfleet job 20261114-165148-pb265-review-r1, at e78b02d) raised two P0 and two P2 findings. 23e4a22 and 93914ff fixed all four.
  • Round 2 (Subfleet job 20261114-190223-pb265-review-r2, at 93914ff) raised no P0 or P1. Its verdict line reads REQUEST_CHANGES, and its highest finding is a P2: no test checked the introduction's claim about the other 11 Claude models. 0fd59e7 adds that test and fixes the round's three P3 wording notes.
  • Checks pass at 0fd59e7.

Round 2 also noted, outside this PR, that the first table gives Claude Sonnet 5 one cost where the other two rows give the board's cost and the run's. I'll check that row separately.

@MaxGhenis
MaxGhenis marked this pull request as ready for review November 15, 2026 04:00
@MaxGhenis
MaxGhenis merged commit 34f0b76 into main Nov 15, 2026
5 checks passed
@MaxGhenis
MaxGhenis deleted the august-note-litellm-correction branch November 15, 2026 04:01

This branch was successfully deployed

1 active deployment
Preview — 0fd59e7e Deployed Nov 15, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant