Skip to content

chore(docs): search and answer engines (#287) - #349

Merged
baldurpan merged 10 commits into
mainfrom
chore/287-search-and-answer-engines
Nov 10, 2026
Merged

baldurpan merged 10 commits into
mainfrom
chore/287-search-and-answer-engines

Conversation

@baldurpan

Copy link
Copy Markdown
Contributor

Closes #287

Ports what worktree's docs still lacked from northguild/gmt#478 for search engines and AI answer engines.

Phases landed (one commit each):

  1. Sitemap <lastmod> from each page's footer date (a0bf0f6)
  2. Long caching for hashed assets, noindex for text files, via _headers (d4ab279)
  3. llms.txt Optional section for the changelog, and the describedby link from each page (08739c5)
  4. Audit fails on a built page over 1 MiB (f96ddf3)
  5. Core Web Vitals measuring script (3d62eba)
  6. Search engine owner steps in docs/README.md (c2707b8)

Release note: none owed. The branch touches only docs/**, context/** and pnpm-lock.yaml, which context/release.md marks as owing no note. The changes go live with the next stable release, since a docs-only merge causes none.

Owner checks left for a person (not defects): #346 (live edge applies _headers), #347 (submit sitemap, add Sitemap: line), #348 (cut a release). The live PageSpeed finding is #345.

Phase 1 of #287. Register @astrojs/sitemap explicitly with a serialize
hook that reads each built page's footer <time datetime> as its
<lastmod> (D1); the landing, which has no footer, gets none (D2).
audit-dist fails a missing, extra, differing or non-W3C <lastmod>.

Gate 1: pnpm check, typecheck (root and docs), build, docs:build,
worker:bundle-check, docs:audit-dist ("28 pages and the site checks
pass"), pnpm test 3826/3826, docs:test 764/764, test markers clean.
sitemap-0.xml: 28 <loc>, 27 <lastmod>, none on the landing.
Gate 2: reviewer subagent, FAIL (type assertions) then PASS WITH NOTES
after one loopback; notes in context/notes.md.
Phase 2 of #287. docs/deploy/_headers gives /worktree/_astro/* a year's
immutable Cache-Control and /worktree/*.txt X-Robots-Tag: noindex;
stage:deploy copies it to the assets root (D4-D6). The audit fails any
_astro/ file without a hash segment in its name, and deploy-smoke checks
the exact header values plus a page carrying neither (D7). The audit
fixture's 3-character hash names are lengthened to 6 because the new
rule would otherwise fail them; no assertion changed.

Gate 1: pnpm check, typecheck (root and docs), build, docs:build,
worker:bundle-check, docs:audit-dist, pnpm test, docs:test, test
markers, all exit 0. stage:deploy served by wrangler dev --local:
Cache-Control immutable on _astro, noindex on llms.txt, neither on a
page; deploy:smoke "every row passed (attempt 1)" (plan Q2: yes).
Gate 2: reviewer subagent PASS WITH NOTES; the host then pinned the
page's revalidation in the smoke row and renamed a missed fixture,
re-reviewed PASS WITH NOTES. Remaining notes in context/notes.md.
…each page (#287)

Phase 3 of #287. renderLlmsTxt emits the 27 documentation pages under
## Documentation and the changelog under ## Optional, the llmstxt.org
section for skippable links (D9). Head.astro adds
<link rel="describedby" type="text/plain"> to the absolute llms.txt URL,
exported from llms-txt.ts, on every page but the 404 (D8). The audit
fails a misplaced section item, a missing, doubled or wrong link, a link
whose target is not built, and a link on the 404.

Gate 1: pnpm check (35 warnings; the new one is Biome not seeing the
Astro template's use of LLMS_TXT_URL), typecheck (root and docs), build,
docs:build, worker:bundle-check, docs:audit-dist, pnpm test, docs:test
780, test markers, all exit 0. dist: 28/28 pages carry one link, 404.html
none, changelog alone under ## Optional.
Gate 2: reviewer subagent PASS WITH NOTES; notes in context/notes.md.
Phase 4 of #287. audit-dist fails any built index.html or 404.html over
PAGE_SIZE_LIMIT (1,048,576 bytes uncompressed, D10) and names the page
and its size. Source: Apple TN3156, "The main resource located at the
previewed link is limited to 1 MB" (read from Apple's page JSON; it does
not say 10^6 or 2^20 bytes, and D10's 2^20 is kept).

Gate 1: pnpm check, typecheck (root and docs), build, docs:build,
worker:bundle-check, docs:audit-dist, pnpm test, docs:test 783, test
markers, all exit 0. Largest built page: commands/branch/ at 178,908 B.
Gate 2: reviewer subagent PASS WITH NOTES; notes in context/notes.md.
Phase 5 of #287, the script part. docs/scripts/web-vitals.mjs measures
the landing, a guide and commands/branch/: Lighthouse 13.6.0 through
npx (mobile and desktop, median of 3: LCP, CLS, TBT), INP in Playwright
Chromium at 4x CPU over search, the theme toggle and the mobile menu,
and PageSpeed Insights read-only for a live origin (PSI_API_KEY from the
environment only). Adds playwright as a docs devDependency (D11); it
downloads no browser on install. Not a gate; writes no numbers.

Gate 1: pnpm check, typecheck (root and docs), build, docs:build,
worker:bundle-check, docs:audit-dist, pnpm test, docs:test 783, test
markers, all exit 0. Gate 2: reviewer subagent PASS WITH NOTES; notes in
context/notes.md. The live run is still owed (PSI quota, sandboxed
Chromium), so the phase is not done.
Phase 6 of #287. A Search engines section in docs/README.md: submit
sitemap-index.xml to Google Search Console and Bing Webmaster Tools, read
each Pages report, what a healthy result is (28 discovered and indexed, no
Duplicate or Alternate page entries), a live header spot-check, and the
root robots.txt Sitemap: line as living in the northguild.dev repository.

Gate 1: docs-only, so Lint (pnpm check, exit 0) plus a read of the diff.
Gate 2: reviewer subagent, PASS WITH NOTES, no blocking items; one wording
note applied, the rest in context/notes.md.
@baldurpan baldurpan added the docs-site The Astro/Starlight site at northguild.dev/worktree label Nov 10, 2026

@baldurs-reviewer baldurs-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code-reviewer: minor issues

This change adds <lastmod> to the sitemap from each page's built footer date, and a _headers file giving _astro/* a year's immutable caching and *.txt a noindex. It also splits llms.txt into Documentation and Optional sections, adds a describedby link on every page, adds audit rules for hashed asset names and page size, adds new smoke-test header rows, and adds a by-hand Web Vitals script. The code is careful and well tested, and I found no correctness bugs in the audit, sitemap or llms.txt logic. My one concern is that the new smoke rows make the release's docs deploy depend on edge behaviour that has not been seen live yet (#346). docs-deploy.yml is not in the diff, so I assumed its site job stages through pnpm --filter docs stage:deploy rather than its own cp. If it copies by hand, _headers is never deployed, and the new smoke rows fail every release.


Reviewed 9f0e3f8 for baldurpan · claude-opus-5-5 · 17 files · left out: pnpm-lock.yaml (lockfile or generated) · $0.66

Comment thread docs/scripts/deploy-smoke.mjs Resolved
Brings in the /dox page (#344). Seven paths conflicted:

- docs/astro.config.mjs: both sides registered @astrojs/sitemap; one call now takes
  main's `filter` (keeps /dox/ out) and this branch's `serialize` (lastmod from the footer).
- docs/package.json: main's exact pin of @astrojs/sitemap.
- pnpm-lock.yaml: main's, regenerated from the merged manifests.
- docs/scripts/audit-dist.mjs: both doc-comment paragraphs.
- docs/scripts/audit-dist.test.mjs: this branch's lastmod-aware sitemap fixture and main's
  Dox fixtures; the chat-launcher fixtures and tests go, as on main; main's three-character
  fixture hashes lengthened for this branch's `_astro/` hash rule.
- context/stack.md, context/verify.md: both sides' edits.
…287)

The merge of main brought auditDox, which calls sitemapUrls; this branch replaced that function with sitemapEntries, which reads lastmod too. The bare catch around the call swallowed the ReferenceError, so the audit passed while never checking that /dox/ is kept out of the sitemap. The test 'fails a sitemap that lists /dox/' caught it.
@baldurpan

Copy link
Copy Markdown
Contributor Author

Merged main (21c1788, the /dox page, #344) into this branch: 4a1d345, plus one follow-up, cc0d147.

The call: docs/astro.config.mjs, overlapping logic. Both sides registered @astrojs/sitemap: this branch before starlight() with serialize (each entry's lastmod read from its built footer, D1/D2), and main after react() with filter (keeps /dox/ out, #309 D6). Git merged both without a conflict marker, which would have registered the integration twice. There is now one sitemap({ filter, serialize }), in main's position. Starlight finds an existing sitemap by name across every integration (@astrojs/starlight/dist/index.js:55), so the position doesn't affect anything. The built sitemap has 28 entries, 27 lastmods (none on the landing) and no /dox/.

Path What was done
docs/astro.config.mjs the single sitemap() above; main's site comment
docs/package.json main's exact pin, "@astrojs/sitemap": "3.7.4"; this branch's playwright, web-vitals and stage:deploy kept
pnpm-lock.yaml main's, regenerated with pnpm install --lockfile-only; it now differs from main only by playwright
docs/scripts/audit-dist.mjs both doc-comment paragraphs (the _astro/ hash rule and the Dox rules)
docs/scripts/audit-dist.test.mjs this branch's lastmod-aware sitemap() fixture with main's Dox fixtures; the chat-launcher fixtures and the four chat-chunk tests go, as on main, which retired the launcher and the rule. Main's three-character fixture hashes (DoxPage.mno.js, dox.pqr.css, page.abc.js) are lengthened, since this branch's rule fails any _astro/ file with a hash under five characters
context/stack.md this branch's wrangler.jsonc line (_headers, web-vitals.mjs), main's docs/worker/ line (/api/brains, [ai], DOX_USAGE)
context/verify.md main's accessible-names sentence (no launcher shortcut) with this branch's hash rule appended; the #287 Phase 3, Phase 4 and lastmod paragraphs and main's #309 paragraph all kept

cc0d147, where the branch had to catch up. Main's new auditDox calls sitemapUrls(), and this branch had replaced that function with sitemapEntries(), which also reads lastmod. The call sits inside a bare catch {}, so the ReferenceError was swallowed: pnpm docs:audit-dist still exited 0, but it would never have flagged /dox/ in the sitemap. Main's test fails a sitemap that lists /dox/ is what caught it. The follow-up reads through sitemapEntries(). That bare catch will hide the next mistake of this kind just as quietly.

Verified: every line each side added since the merge base is in the result, apart from the deliberate removals above, checked per path. All of context/verify.md exits 0 on cc0d147's tree: pnpm check, both typechecks, pnpm build, pnpm docs:build, worker:bundle-check, pnpm docs:audit-dist (28 pages), pnpm test (3826), pnpm docs:test (1308) and the test-marker check.

@baldurs-reviewer baldurs-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code-reviewer follow-up: minor issues

Since the last review the branch merged main, which brought in #309's Dox page and the Workers AI brains. Within this PR's files that means: the sitemap integration moves after Starlight and gains a filter that drops /worktree/dox/; @astrojs/sitemap is pinned; the audit drops the launcher and shortcut rules for auditDox, header and mobile-menu Dox-link rules, and an "Ask Dox" link rule; and docs and context are updated to match. The new audit rules are consistent with their tests, and I found no new defect. Two stale descriptions sit outside the hunks: docs/README.md line 266 still lists the header's right group as theme, language, GitHub, Discord (without Dox), and the header comment of audit-dist.mjs (lines 8–9) still says the audit checks for a script that loads the chat before it is activated. My earlier comment on the deploy-smoke header rows still stands, since that file is unchanged.

Earlier comments


Reviewed cc0d147 (changes since 9f0e3f8) for baldurpan · claude-opus-5-5 · 7 files · left out: pnpm-lock.yaml (lockfile or generated) · $0.59

The merge of main (#309) added the "Chat with Dox" link to the header's right group and retired the chat launcher, and two descriptions in files this branch edits still said otherwise: docs/README.md listed the right group without Dox, and the audit's header comment still named the launcher-era chat-script rule. Raised by baldurs-reviewer on #349.

@baldurs-reviewer baldurs-reviewer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

code-reviewer follow-up: minor issues

Since the last review, two lines changed, both in documentation. docs/README.md now lists the "Chat with Dox" crystal first in the header's right group and counts four controls. The header comment in audit-dist.mjs now describes the Dox-code rule in place of the retired "chat before activation" rule. Both match the code they describe and raise nothing new. The earlier risk about the new smoke-check header rows in deploy-smoke.mjs is unchanged and still open.

Earlier comments


Reviewed f9c795f (changes since cc0d147) for baldurpan · claude-opus-5-5 · 2 files · $0.29

@baldurpan

Copy link
Copy Markdown
Contributor Author

The two stale descriptions from the follow-up review (no thread to reply in) are fixed in f9c795f:

  • docs/README.md:266: the right group now reads the "Chat with Dox" crystal, the theme button, the language select, then GitHub and Discord, as four controls of one size. That matches Header.astro's own doc comment and the Opening it section of the same file.
  • docs/scripts/audit-dist.mjs:9: "no script that loads the chat before it is activated" is now "no Dox code, which loads on /dox/ alone (dox-page #309, below)". The launcher-era rule went with the merge.

Both lines were already stale on main after #344, but both files are in this PR and it was the merge here that made them untrue, so they're fixed here. Prose and a comment only, so pnpm check exits 0. The deploy-smoke thread has its own reply.

@baldurpan
baldurpan merged commit 66c63f7 into main Nov 10, 2026
1 check passed
@baldurpan
baldurpan deleted the chore/287-search-and-answer-engines branch November 10, 2026 22:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs-site The Astro/Starlight site at northguild.dev/worktree

Projects

None yet

Development

Successfully merging this pull request may close these issues.

search-and-answer-engines

1 participant