Skip to content

Sitemap, logo and Organization data for search engines - #21

Merged
craig-o-curtis merged 2 commits into
mainfrom
dox-g6-landing-seo
Nov 7, 2026
Merged

craig-o-curtis merged 2 commits into
mainfrom
dox-g6-landing-seo

Conversation

@craig-o-curtis

@craig-o-curtis craig-o-curtis commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Part of northguild/gmt#448 (DOX-G6). The /gmt/ docs changes for that issue are in their own PR in the gmt repo; this PR covers the landing page only.

What changes

  • Sitemap. @astrojs/sitemap 3.7.4 (pinned) writes sitemap-index.xml and sitemap-0.xml, which lists https://northguild.dev/. The package leaves the 404 page out itself. No lastmod, since the site has no per-page date that is verifiably accurate.
  • robots.txt. A second line, Sitemap: https://northguild.dev/sitemap-index.xml, goes before the /gmt/ one, and Disallow: /gmt/api/ keeps crawlers off the docs chat API (JSON endpoints only, no pages). Every other line is unchanged, Content-Signal: search=yes, ai-input=yes, ai-train=yes included.
  • Logo. public/logo.png, 512×512, rendered from favicon.svg. To re-run: rsvg-convert -w 512 -h 512 -f png -o public/logo.png public/favicon.svg (librsvg 2.63.2), from apps/site. The mark keeps its own dark tile, so it shows on white.
  • Organization data. The home page writes one schema.org Organization JSON-LD block: name, url, logo and sameAs (https://github.com/northguild). < is escaped as \u003c, so a value cannot close the script tag.
  • 404 page. noindex, and no canonical link. Before, /nope answered 404 with a canonical pointing at /404/, which answered 200.
  • aurora.worker.ts. setTimeout becomes self.setTimeout, as the file already does elsewhere. The sitemap package brings in @types/node, which typed a bare setTimeout as Node's Timeout and failed pnpm check. No change in the browser. The same dependency changes the astro, vite and sharp keys in pnpm-lock.yaml.

Checks

  • pnpm lint, pnpm format:check, pnpm check and pnpm build pass.
  • In dist/: the sitemap lists / and no 404; robots.txt has both Sitemap: lines; logo.png is a 512×512 PNG; index.html has exactly one JSON-LD block, @type Organization; 404.html has noindex, no canonical and no JSON-LD.
  • Under wrangler dev: /sitemap-index.xml and /logo.png answer 200; /nope answers 404 with noindex.
  • / at 1440 and 390 looks the same before and after. The change is in <head> only.

Owner steps after deploy

  • Submit https://northguild.dev/sitemap-index.xml in Google Search Console and Bing Webmaster Tools.
  • Check https://northguild.dev/ in Google's Rich Results Test and at validator.schema.org. Both should read the Organization with no errors.

The home page is now in a sitemap, robots.txt names that sitemap
beside the /gmt/ one, and the docs chat API under /gmt/api/ is no
longer crawled. Every original robots.txt line stays, Content-Signal
with ai-train=yes included.

The home page states schema.org Organization data (name, url, logo,
sameAs) and serves a 512 px logo.png rendered from favicon.svg. The
404 page carries noindex and no canonical, so it cannot be indexed
as /404/.

aurora.worker.ts calls self.setTimeout: the sitemap package brings
in Node types, which made a bare setTimeout return Node's Timeout.

Refs northguild/gmt#448

@baldurpan baldurpan left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read the diff, plus the @astrojs/sitemap 3.7.4 source and the Dox worker's routes in the gmt repo. It does what the description says. /gmt/api/ is where the Dox worker answers chat, brains and the clock-change checks. logo.png is a 512×512 render of the favicon tile. The \u003c escape stops a value from closing the script tag. self.setTimeout matches the two calls further down the file.

I left two comments, and neither blocks the merge. One is about the repo's own rule on robots.txt. The other is a filter that the sitemap package already applies itself.

Edited: this review first said the description had lost a backslash in the escape. That was wrong: the description said \u003c all along, and it was my own text that lost it. See my reply on astro.config.mjs.

Comment thread apps/site/public/robots.txt Resolved
Comment thread apps/site/astro.config.mjs Resolved
…txt rule

@astrojs/sitemap leaves 404 and 500 pages out before it calls a custom
filter, so the filter never removed anything; sitemap() alone gives the
same sitemap.

The CLAUDE.md rule on robots.txt said to keep the file exactly as is,
which the sitemap and /gmt/api/ lines now break. It now says to keep
every existing line, Content-Signal included, and to add a line only
for a new sitemap or a path crawlers should skip.

Refs northguild/gmt#448

@baldurpan baldurpan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved at a073639. Both comments are fixed: CLAUDE.md now says what in robots.txt is fixed and when to add a line, and sitemap() no longer carries a filter that never removed anything. CI is green. Thanks for the quick turnaround, and sorry for the wrong note about the description.

@craig-o-curtis
craig-o-curtis merged commit 8f94828 into main Nov 7, 2026
1 check passed
@craig-o-curtis
craig-o-curtis deleted the dox-g6-landing-seo branch November 7, 2026 19:23
@craig-o-curtis

Copy link
Copy Markdown
Contributor Author

Owner check after deploy: https://northguild.dev/ reads one Organization (name, url, logo, sameAs) with 0 errors and 0 warnings.

craig-o-curtis added a commit to northguild/gmt that referenced this pull request Nov 8, 2026
…X-G6) (#478)

Closes #448
Closes #456

New OG images:

<img width="819" height="424" alt="Image"
src="https://github.com/user-attachments/assets/bce3fb5e-d188-44d6-9ae9-e4a778685473"
/>

Method pages og images:

<img width="849" height="436" alt="Image"
src="https://github.com/user-attachments/assets/34374576-5bdb-44ea-85ee-a22d477a4596"
/>

Makes `northguild.dev/gmt/` easier for search engines and AI answer
engines to crawl, index and quote. Everything follows documented
mechanics: crawlable text, accurate sitemaps, one indexable URL per
page, lighter pages, and an `llms.txt` that fits in an agent's context.
AI crawlers stay allowed. No `packages/gmt` change and no changeset.

## Search descriptions

- No two indexable pages share a meta description. A reference
function's description is its TSDoc summary. Where sibling functions
share one, a qualifier names the value kind or the namespace.
- Each namespace has one written summary. It is both the namespace
page's meta description and its line in `llms.txt`, and it replaces 13
identical boilerplate descriptions.
- A test on the built site fails when two indexable pages share a
description, and lists them.

## Crawling and `llms.txt`

- Generated pages carry a real Last updated date from the newest commit
of their inputs. The sitemap's `<lastmod>` is copied from that footer
date, so the two agree.
- A tracked `_headers` file caches hashed `/gmt/_astro/*` files for a
year as immutable, and marks the `.md` twins and `llms*.txt` files
`noindex`.
- The 404 page carries `noindex` and no canonical.
- The root `/gmt/llms.txt` fits in 64 KiB (it was about 213 KB). Its
Reference section links one `llms.txt` per namespace. Every page carries
`rel="describedby"` pointing at its namespace file, or at the root file
outside the reference.
- Each page keeps its own namespace's tree in the sidebar, and every
other namespace shows one collapsed Overview link. Page HTML is 33% to
47% smaller on the four pages measured, and previous and next links are
unchanged.
- `deploy-smoke.mjs` checks the crawl rows after each deploy.

## Share cards and tags

- Every share card shows the Earth, with its text inside the centre
square that link-preview apps crop to. Cards stay under 300 KB. No card
text is cut with an ellipsis; a title that does not fit fails the build.
- Share tags: `og:locale` is `en_US`, `og:image:type` is set,
`og:image:alt` and `twitter:image:alt` describe the card, and both image
URLs end in `?v=<hash of the PNG>` so a redrawn card is a new URL.
- A test on the built site checks every indexable page for exactly one
of each tag, equal image URLs and a hash that matches the PNG.
- Upload-ready copies of the cards are in `apps/dox/assets/share/`.

## Changelog pages

- The changelog is one page per release line (`/changelog/1-21/`) plus
an index, so no page nears 1 MB. Apple's Messages preview reads only the
first 1 MB of a page, and the single page was 1,546,542 bytes. The
largest page is now 729,975 bytes, and a build test fails on any page
over 1,048,576 bytes.

## Root site card (other repo)

The landing page `northguild.dev/` is a separate site. Its search data
and card are made in `northguild/northguild.dev`, and both PRs are
merged:

- northguild/northguild.dev#21: sitemap, logo and Organization data.
- northguild/northguild.dev#22: a card that survives a square crop,
under 300 KB, with a versioned image URL.

## Globe, LCP and INP

- The home globe starts when the page is idle and sets up in slices that
yield to the main thread, so no task runs long.
- Every globe pauses while off screen, while the tab is hidden, and
while the search dialog or the mobile menu covers it.
- Where the globe falls back to canvas 2D, it draws in a module Web
Worker on an `OffscreenCanvas`. Input no longer waits for a globe frame.
A probe compares text rasterisation on both paths first; if they differ,
or the worker fails, the main-thread renderer takes over. The look is
unchanged.
- The hero clock loads `@northguild/gmt` by dynamic import beside the
star list, which keeps the Temporal polyfill off the first-paint path.
- The clock list measures its rows in one pass after writing them.
- `apps/dox/scripts/web-vitals.mjs` measures LCP, CLS, TBT and INP by
hand.

Local lab numbers for the home page on mobile, before then after: TBT
464 ms to 149 ms; LCP 3.0 s (varying between runs) to 2.26 s; INP at 4x
CPU throttling 256 ms to 96 ms with WebGPU, and 392 ms to 96 ms with
canvas 2D. Guide and reference pages were already good.

## Owner steps

From `context/domination/issues/CORE-86/search-and-crawlers.md` (runbook
§8).

- [x] Rich Results Test on `https://northguild.dev/`
(northguild/northguild.dev#21): Organization, 0 errors, 0 warnings.
- [ ] **O1** Search Console: submit both sitemaps and record the Pages
report.
- [ ] **O2** Search Console URL Inspection on `/gmt/` and two reference
pages.
- [ ] **O3** Bing Webmaster Tools: import from Search Console, then
submit the sitemaps.
- [ ] **O4** AI Crawl Control: record the 30-day request counts per
crawler.
- [ ] **O5** Crawler Hints: turn it on. Turn it off if Bing lists no
IndexNow URLs within a week.

## Open checks after deploy

These run against the live site, so they stay open. The list is also in
`context/dox/built.md`.

- [ ] Facebook Sharing Debugger, LinkedIn Post Inspector, X, Slack,
Discord, WhatsApp: the card shows for `/gmt/`, a guide, a reference page
and `/gmt/changelog/`; WhatsApp shows the whole title in a square crop.
- [ ] iMessage: the rich preview shows for the same URLs, including a
changelog line page.
- [ ] The same platform checks for `https://northguild.dev/`.
- [ ] `node apps/dox/scripts/deploy-smoke.mjs` passes against
`https://northguild.dev`.
- [ ] Rich Results Test and validator.schema.org read the Organization
data on `https://northguild.dev/`.
- [ ] `node scripts/web-vitals.mjs --live https://northguild.dev` with
`PSI_API_KEY` set records PageSpeed Insights and CrUX numbers.

## Baseline

Search Console, Bing and AI Crawl Control numbers from before deploy:

_To be filled in by the owner._

## Validation

- Each slice was checked by a tester against its definition of done:
- Search, crawl and `llms.txt`: the Dox Vitest suite, every built-site
test on a fresh build, `astro check`, lint, a local Wrangler run of
`deploy-smoke.mjs`, and a whole-site HTML comparison against `main`'s
build.
- Lean sidebar: the browser and keyboard checks, and the HTML sizes
listed above.
- Share cards and tags: card unit tests (text inside the square,
contrast at least 4.5:1, no ellipsis) and `share-preview.test.ts` on a
fresh build.
  - Changelog pages: the size test on a fresh build.
- Globe: pixel comparison of the worker against the main-thread
renderer, and the pause rules, in Chromium. Interior pixels differ by 1
to 14 levels where the ocean pattern is filled, and the clock card can
show one frame before the globe when it mounts. Both are accepted.
- Every test that reads the built site now runs in CI, found by rule
(`apps/dox/scripts/dist-tests.mjs`). All 11 files pass on a fresh build.
- CI on 9a3fee6: every job passed, including the Dox gates. The review
fixes are in d516fc3.
- Local Lighthouse, mobile, home page: performance 78 to 84 on `main`,
93 to 95 here.
- `pnpm deps:sync`: no change to any `Blocked by` cell.

Not verified:

- `verdict validate` has not run locally: the scratch copy it would run
in is not a git checkout. CI runs the same build, lint and test steps.
- The globe worker has not run in real Firefox or Safari. Firefox would
not start on the test machine.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants