Verify what AI writes about science — citations, claims, and cross-file consistency.
A README linked to PMID 12078741 as the foundational paper on Restricted Focus Viewers in vision science. The actual paper at that ID? "Determination of true ileal amino acid digestibility... in barley samples for growing-finishing pigs." The correct PMID was 12723780 — off by 645,039.
npx github:andyed/science-agent audit ./docs --bibtex=./refs.bibaudit verifies every DOI-bearing BibTeX entry against Crossref by default,
then compares canonical title, complete author list, publication year, volume,
and pages. Use --offline only when you intentionally want the local
author/year-to-BibTeX scan without that metadata guarantee.
Set SCIENCE_AGENT_CROSSREF_MAILTO if you want requests to opt into
Crossref's contact-identified polite pool. No email address is sent by default.
Or verify a single DOI against CrossRef:
npx github:andyed/science-agent verify 10.1038/nn.2889No install, no clone — runs straight from this repo.
Want it on your $PATH instead of refetching each run? Install the CLI globally from GitHub (wires up the science-agent bin):
npm install -g github:andyed/science-agent
science-agent audit ./docs --bibtex=./refs.bibCatches AI-confabulated academic citations before they ship. Verifies inline references against BibTeX and CrossRef:
| Pattern | How |
|---|---|
| Wrong title | Fuzzy title matching against BibTeX + CrossRef |
| Fabricated co-authors | CrossRef author list verification |
| Wrong DOI | CrossRef DOI resolution — checks that the DOI points to the claimed paper |
| Compound confabulation | CrossRef + title search detects merged citations |
| Ambiguous citation | Surname+year collision detection across BibTeX entries |
| Duplicate citation key | A key defined more than once keeps its first entry, as BibTeX does; the repeat is reported (duplicate_key), as an error when the entries differ |
| Orphan citation | Inline reference with no BibTeX entry |
Research notebooks produce numbers. Papers and READMEs cite those numbers. Over time, numbers drift — a notebook gets re-run with new data, but the prose still quotes the old value. Science Agent makes this auditable.
The idea: Each notebook declares its load-bearing results in a ## Key Claims table. Prose references them as [NB14:K3] (notebook 14, claim 3). Science Agent verifies every reference resolves to a real claim with a real value.
## Key Claims
| ID | Claim | Value | Verified |
|----|-------|-------|----------|
| K1 | Sample size after exclusions | N = 2,719 trials | 2026-04-09 |
| K2 | Main effect | ρ = −0.618, p = 0.0426 | 2026-04-09 |Then in your paper or README:
The position × cognitive load correlation [NB14:K2] suggests...
Audit it:
# Generate aggregate from notebooks (one-time setup)
science-agent aggregate ./notebooks/ -o docs/notebook-key-claims.md
# Audit all claim references in prose
science-agent notebook-audit ./docs \
--aggregate=./docs/notebook-key-claims.md \
--notebooks=./notebooks/ \
--cross-repo=../downstream-repoDetects:
- Dangling references —
[NB14:K3]cited in prose but K3 doesn't exist in NB14 - Missing Key Claims blocks — notebook is cited but has no auditable claims table
- Stale cross-repo values — downstream repo quotes pre-fix numbers from upstream
If you don't use notebooks or don't need claim tracking, ignore this — audit and verify work standalone. If you do, see the full setup guide: docs/notebook-conventions.md
A figure gets regenerated; its caption keeps the old numbers. figure-audit reads an INDEX.md, finds each figure's summary.json sidecar, extracts the numerics from the caption prose, and reports each as matched, unverified, or mismatched against the stats dump.
science-agent figure-audit ./figures/INDEX.mdThe audits above are consumer-side: they check prose against what a producer wrote. None of them sees a script that computes a new number on a code path that can no longer reproduce its already-published one. The prose then quotes a wrong value faithfully and every other audit passes.
A gated producer recomputes a shipped number on the same inputs before reporting anything new, aborts if it does not come back, and records the outcome in its summary.json:
"gate": {
"status": "ok",
"shipped": 0.691, "reproduced": 0.691, "delta": 0.0, "tolerance": 5e-4,
"source": "scripts/output/m4_cursor_only_downstream/summary.json",
"source_path": "section_4_3.deployable_M4_7.pooled_auc"
}repro-audit reads those blocks across an output tree:
science-agent repro-audit ./scripts/output
science-agent repro-audit ./scripts/output --require-gate # every producer must be gated| Finding | Severity | Meaning |
|---|---|---|
gate_failed |
error | Delta exceeds tolerance. The comparison is recomputed, so a block claiming ok with a blown delta still fires |
stale_pin |
error | The gate passes, but the cited source no longer carries the pinned value |
gate_skipped_undocumented |
error | Skipped with no stated reason |
ungated |
warn (error under --require-gate) |
Numbers reported with no gate |
uncited_pin |
warn | Pinned to a literal instead of source + source_path, so staleness cannot be checked |
A run that scans zero sidecars exits 2 instead of reporting a clean result. Full contract: agents/repro-gate.md.
prose-audit lints .md, .tex, and .ipynb drafts for patterns that mark machine-written prose: significance inflation, copula avoidance, sentence fragments, em-dash and semicolon density, repeated phrases. One rule is a rigor check: weasel-attribution fires at error when an authority phrase (studies show, it is widely believed) has no citation in the same paragraph.
science-agent prose-audit ./paper --summaryRules live in src/aiism-rules.json and run in Node with no Python dependency. Each rule has must_fire and must_not_fire fixtures in test-fixtures/prose/.
Science Agent is a CLI tool — any assistant that can run shell commands can use it. No API keys, no plugins, no vendor lock-in.
Claude Code:
> check my citations against refs.bib
# Claude runs: npx github:andyed/science-agent audit ./docs --bibtex=./refs.bib
ChatGPT / Codex / GitHub Copilot in terminal:
> run science-agent to verify the citations in my paper
# GPT runs: npx github:andyed/science-agent audit ./paper --bibtex=./references.bib
Gemini Code Assist / Cursor / Windsurf / any terminal AI:
> audit my bibtex citations for confabulation
# Assistant runs: npx github:andyed/science-agent audit . --bibtex=./refs.bib
The pattern is the same everywhere: point at a directory of prose and a BibTeX file. The tool does the rest.
For deeper integration, see agent.md (Claude Code agent) or docs/github-actions.md (CI/CD).
For any agent that supports skills.sh — Claude Code, Cursor, Codex, and others — install the portable skill straight from GitHub:
npx skills add andyed/science-agentThis installs skills/science-agent/SKILL.md, which teaches the agent when and how to drive the CLI (audit, verify, search, arxiv-search, notebook-audit, figure-audit, repro-audit, prose-audit) plus the find→verify pattern. The CLI runs via npx github:andyed/science-agent, so there's nothing else to install.
Install the whole toolkit — agents, slash commands, CLI — in one step:
/plugin install andyed/science-agent
This pulls the repo and registers the following. There is no install step: the CLI has no npm dependencies and runs from the plugin directory on Node ≥18.
- Slash commands:
/science-agent:audit,/science-agent:figure-audit,/science-agent:repro-audit,/science-agent:notebook-audit,/science-agent:prose-audit,/science-agent:verify,/science-agent:search,/science-agent:aggregate,/science-agent:arxiv,/science-agent:arxiv-search - Sub-agents (auto-invoked when relevant):
figure-audit,rigor-audit,repro-gate
prose-audit runs a native Node engine — its rule set lives in src/aiism-rules.json (first-party threshold rules plus MIT-compatible ported rules) with detectors in src/prose-audit.js, so it ships in the plugin with no Python dependency. An optional muriel (Python) pass adds extra rules when python3 + muriel are present; pass --no-muriel to skip it, or --no-native to skip the built-in rules.
A planned claim-audit agent will cover semantic claim correctness (does the cited paper actually say this?). It slots into agents/ when Phase 4 of PLAN.md ships.
If you don't want the full plugin, you can drop individual agent files into .claude/agents/:
mkdir -p .claude/agents
curl -o .claude/agents/science-agent.md https://raw.githubusercontent.com/andyed/science-agent/main/agent.md
curl -o .claude/agents/rigor-audit.md https://raw.githubusercontent.com/andyed/science-agent/main/agents/rigor-audit.md
curl -o .claude/agents/figure-audit.md https://raw.githubusercontent.com/andyed/science-agent/main/agents/figure-audit.md
curl -o .claude/agents/repro-gate.md https://raw.githubusercontent.com/andyed/science-agent/main/agents/repro-gate.mdThe CLI must be available on $PATH (or via node cli.js) for figure-audit and repro-gate to work — the plugin path handles this automatically.
# Clone and run. There are no npm dependencies, so there is nothing to install.
git clone https://github.com/andyed/science-agent.git
cd science-agent
# Audit citations and verify every DOI-bearing BibTeX entry against Crossref
node cli.js audit ./docs/specs --bibtex=./refs.bib
# Explicit local-only mode (does not validate BibTeX metadata)
node cli.js audit ./docs/specs --bibtex=./refs.bib --offline
# Audit notebook claim references
node cli.js notebook-audit ./docs --aggregate=./docs/notebook-key-claims.md
# Generate a Key Claims aggregate from notebooks
node cli.js aggregate ./notebooks/ -o docs/notebook-key-claims.md
# Audit recent arXiv papers (baseline check)
node cli.js arxiv 10 --cat=cs.AI
# Verify a single DOI against CrossRef
node cli.js verify 10.1038/nn.2889
# Search CrossRef by title
node cli.js search "Metamers of the ventral stream"
# Search arXiv; surfaces a preprint's published DOI ready to verify
node cli.js arxiv-search "peripheral vision crowding" --max=5
# Verify figure-caption numerics against summary.json sidecars
node cli.js figure-audit ./figures/INDEX.md
# Check that producers reproduced a shipped number before reporting a new one
node cli.js repro-audit ./scripts/output
# Lint drafts for AI-tell prose
node cli.js prose-audit ./paper --summaryEvery command takes --json and prints exactly one JSON document to stdout. node cli.js --help lists all flags.
| Code | Meaning |
|---|---|
| 0 | Ran and found nothing at the failing severity |
| 1 | Findings at error severity (prose-audit --severity= moves the bar; default warn) |
| 2 | Nothing was checked: a path that does not exist, or a repro-audit run that scanned zero sidecars |
A missing path never exits 0. audit and aggregate predate the 1/2 split and report it as 1.
audit also exits 1, with no report, when the --bibtex file has a syntax error or holds no entries. The message gives file:line:column and the entry. A bibliography read in part would let the audit pass on the entries that happened to parse.
═══ Science Agent Audit ═══
Directory: ./docs/specs
BibTeX: ./refs.bib
Citations: 62
In BibTeX: 33
Orphans: 29
Cites with DOI: 29
Ambiguous: 1
Bib entries: 41
Bib DOI verified: 39/39
Metadata matches: 38
Metadata mismatches: 1
Missing DOI: 2
Unverified DOI: 0
Issues: 33
── Issues ──
⚠ [ambiguous] Pelli & Tillman (2008)
wave3_crowding_validation.md
matches 2 BibTeX entries — disambiguate with DOI or journal
ℹ [orphan] Schwartz (1980)
cmf_mip_derivation.md
has no BibTeX entry
✗ [wrong_doi] maxwell2018scent
refs.bib
DOI resolves to a different paper
AI coding assistants confabulate academic citations at a measurable rate. The model gets 95% right — correct author surname, approximate title, right journal, right year — then fabricates the remaining 5%. This is dangerous because it passes casual review.
| Source | Error rate |
|---|---|
| Human-authored arXiv papers (our spot-check) | 0% |
| Human-written project docs (our audit) | 0–7% |
| AI-assisted project docs (our audit) | 12–24% |
| AI-generated citations across 13 LLMs (GhostCite) | 14–95% |
The fix: require a DOI where one exists, then verify that it resolves to the claimed title and authors. DOI presence alone is not evidence: an adjacent proceedings DOI can resolve successfully to a completely different paper. Science Agent automates the metadata comparison rather than trusting the shape or resolvability of the DOI string.
See FINDINGS.md for the complete audit with methodology, data, patterns, and the 95/5 confabulation taxonomy.
- Claim content verification — detect wrong numbers attributed to real papers, cross-file consistency for shared parameters
- Research corpus index — catalog local PDFs with extracted metadata, track what's been read
- MCP server — tools for any MCP client
| Project | What it does | How science-agent differs |
|---|---|---|
| GhostCite / CiteVerifier | DBLP-based citation title verification | We use CrossRef (broader coverage), verify DOIs + authors + titles, and work as a Claude Code agent |
| CiteAudit | Multi-agent verification pipeline + web service | Not open source. Science-agent is local-first, CLI, and embeds in your dev workflow |
| CiteME | Benchmark: can LLMs identify source papers from excerpts? | Benchmark, not a tool. Different task (retrieval vs. verification) |
| Context Rot | Measures general LLM degradation with context length | Methodology foundation for understanding why hallucination worsens under load |
| Claude Scholar | Full research lifecycle config for Claude Code | Workflow orchestrator with prompt-based citation checking. Science-agent could serve as its verification backend via MCP |
| K-Dense scientific-agent-skills: citation-management | Discovery + formatting — Google Scholar / PubMed MeSH / CrossRef / arXiv / DataCite search, BibTeX deduplication, sorting, validation | Different emphasis, complementary surface. K-Dense covers finding and formatting references. Science-agent covers verifying them and catching cross-file drift in claims that cite them. MIT-licensed sibling worth running alongside |
| google-deepmind/science-skills | 37 agent skills wrapping scientific databases — genomics (AlphaGenome, ClinVar, gnomAD), structural biology (AlphaFold, PDB, Foldseek, PyMOL), cheminformatics (ChEMBL, PubChem), and literature search (arXiv, OpenAlex, Europe PMC, PubMed). Built for Google Antigravity; Apache-2.0 + CC-BY-4.0 | Retrieval, not verification. DeepMind's skills find data and papers; science-agent verifies the citations and claims already written. Complementary — a retrieval agent can call science-agent verify as its check-step. Our arxiv-search reimplements their literature_search_arxiv query in Node (MIT, no Python dependency) and pipes the published DOI straight into verify |
The arms are AI. The microscope is yours.
andre-inter-collab-llc/research-workflow-assistant — Andre Nogueira's open-source Research Workflow Assistant: a VS Code + GitHub Copilot stack of custom agents and MCP servers (PubMed, OpenAlex, Semantic Scholar, Europe PMC, CrossRef, Zotero) for systematic reviews and academic writing. Different domain (biomedical research workflows vs citation verification), same underlying bet: researchers already have VS Code, git, and Markdown — give them an LLM with the right agent scaffolding and they can assemble their own compliant research assistants.
Built after discovering AI-confabulated citations in Scrutinizer, an open-source peripheral vision simulator. A collaborator checked the arxiv reference to his own paper during a meeting and found the title was wrong. The full audit revealed systematic patterns that replicated across a second project in a different domain.
MIT