Repository navigation
Correct the August thinking note's account of litellm's reasoning count - #265
Merged
Merged
Conversation
The note's second finding said litellm "does not populate reasoning-token fields for Anthropic even when thinking blocks are present", and its Claude Haiku 5.5 section said "litellm reports no reasoning tokens for Anthropic". litellm fills the field from a count of the thinking text a response returns. In the note's runs Claude returned its thinking blocks with empty text, so litellm counted 0. - Reword both passages and add a correction section that quotes the earlier sentences. No score, rank or cost changes. - Add a test: each of the note's five runs records completion tokens and 0 reasoning tokens on every row, Claude Fable 5's board rows record no usage, and only the correction section holds the two earlier sentences. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
A review of the correction found four faults, all fixed here. The note said the five runs' responses returned thinking blocks with empty text. The committed predictions keep no thinking blocks, so nothing shows what those responses returned. The note now states the runs' zeros and cites the recorded Claude Haiku 5.5 probe as the example: Anthropic counted 119 of 123 output tokens as thinking, the thinking block came back empty, and litellm counted 0. The note said the harness records Anthropic's count "whenever a response reports one". The harness records a count only when it is a whole number from 0 to the response's output tokens, and a response replayed from an older cache entry keeps litellm's count. The note now says both. The introduction said every Claude row logged zero reasoning tokens. Claude Fable 5's board rows record no usage. The introduction now limits the claim to rows with recorded usage, and the correction quotes the earlier claim. The test fixed whole sentences of prose and a dated heading. It now checks the data (the runs' counts, Claude Fable 5's missing usage, the probe's block and counts), that the note's body cites the probe and reads a 0 as "not counted", that no withdrawn claim returns to the body, and that the correction quotes each withdrawn claim in full under any date. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…e test's gaps Two more readers checked the correction and found no false statement. This commit takes their smaller points. The note: - names Claude Fable 5's missing usage in the introduction, beside the claim it limits, and says the rows are those in the board's predictions file; - gives Anthropic the verb in the probe example and says litellm counts 0 for the recorded response, which the thinking-token test replays; - states what the harness does since #261 for a response with no count (litellm's count on the sync path, none on the batch path) and for a replay from an older cache entry, and leaves the bound to the runbook; - splits the Claude Haiku 5.5 sentence in two; - tightens the correction section and keeps its three quotations. The test: - searches the body for each withdrawn claim without regard to case and with the nearby wordings (the field or fields; no, zero or 0; Anthropic or Claude; logged or recorded); - finds the correction section by its heading, whatever follows the word, and treats any section after it as body; - requires the probe's two counts in order; - documents the check that the correction says no score, rank or cost changed. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…icyBench verified live The test now checks that the 11 Claude models other than Claude Fable 5 record completion tokens and 0 reasoning tokens on every row of the board's predictions, as the note's introduction says. The note's lead-in to its two findings says PolicyBench verified only the first with live requests. The second finding states the bound on Anthropic's count and the cases where the harness still records litellm's count or none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
Review record at 0fd59e7:
Round 2 also noted, outside this PR, that the first table gives Claude Sonnet 5 one cost where the other two rows give the board's cost and the run's. I'll check that row separately. |
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The August thinking note (
sensitivity/claude-thinking-2026-08.md) said litellm "does not populate reasoning-token fields for Anthropic even when thinking blocks are present". Its Claude Haiku 5.5 section said "litellm reports no reasoning tokens for Anthropic", and its introduction said "every Claude row logged zero reasoning tokens". litellm fills the field from a count of the thinking text a response returns, and Anthropic can return a thinking block with empty text. Claude Fable 5's rows in the board's predictions file record no usage. In #261 we found how litellm counts and left the note unchanged.What changes
reasoning_tokensempty for any other reported count. For a response that reports no count, it still records litellm's count on the sync path and none on the batch path. When it replays a response from an older cache entry, it keeps litellm's count. The note tells readers to read a 0 in these runs'reasoning_tokensas not counted.Evidence
reasoning_tokensfrom a token count of the returned thinking text and never read Anthropic'sthinking_tokens(llms/anthropic/chat/transformation.py). The lockfile pinned 1.88.1 for all five of the note's sensitivity runs and has pinned 1.95.0 since November 2.tests/fixtures/anthropic_haiku55_probe_responses.jsonreturns one thinking block with empty text, where Anthropic counts 119 of 123 output tokens as thinking.tests/test_anthropic_thinking_tokens.pyreplays it through litellm, which counts 0._bounded_thinking_tokensinpolicybench/eval_no_tools.pysets the bound on Anthropic's count, anddocs/runbook.mdsection 8b lists the cases that still record litellm's count or none.Test
test_doc_reads_the_runs_zero_reasoning_tokens_as_not_countedchecks the five runs' counts in the committed predictions. In the board's predictions it checks Claude Fable 5's missing usage and the other 11 Claude models' 0 reasoning tokens. It also checks the probe's block and counts in the fixture. It requires the note's body to cite the probe and its two counts, to tell readers to read a 0 as "not counted", and to hold no sentence that matches a withdrawn claim's pattern. It requires the correction section, whatever its date, to quote each withdrawn claim in full and to say that no score, rank or cost changed. It fails on main's note.Local:
tests/test_sensitivity_evidence.pyandtests/test_disclosures.py, 55 passed.🤖 Generated with Claude Code