Repository navigation
Sitemap, logo and Organization data for search engines - #21
Conversation
The home page is now in a sitemap, robots.txt names that sitemap beside the /gmt/ one, and the docs chat API under /gmt/api/ is no longer crawled. Every original robots.txt line stays, Content-Signal with ai-train=yes included. The home page states schema.org Organization data (name, url, logo, sameAs) and serves a 512 px logo.png rendered from favicon.svg. The 404 page carries noindex and no canonical, so it cannot be indexed as /404/. aurora.worker.ts calls self.setTimeout: the sitemap package brings in Node types, which made a bare setTimeout return Node's Timeout. Refs northguild/gmt#448
There was a problem hiding this comment.
I read the diff, plus the @astrojs/sitemap 3.7.4 source and the Dox worker's routes in the gmt repo. It does what the description says. /gmt/api/ is where the Dox worker answers chat, brains and the clock-change checks. logo.png is a 512×512 render of the favicon tile. The \u003c escape stops a value from closing the script tag. self.setTimeout matches the two calls further down the file.
I left two comments, and neither blocks the merge. One is about the repo's own rule on robots.txt. The other is a filter that the sitemap package already applies itself.
Edited: this review first said the description had lost a backslash in the escape. That was wrong: the description said \u003c all along, and it was my own text that lost it. See my reply on astro.config.mjs.
…txt rule @astrojs/sitemap leaves 404 and 500 pages out before it calls a custom filter, so the filter never removed anything; sitemap() alone gives the same sitemap. The CLAUDE.md rule on robots.txt said to keep the file exactly as is, which the sitemap and /gmt/api/ lines now break. It now says to keep every existing line, Content-Signal included, and to add a line only for a new sitemap or a path crawlers should skip. Refs northguild/gmt#448
baldurpan
left a comment
There was a problem hiding this comment.
Approved at a073639. Both comments are fixed: CLAUDE.md now says what in robots.txt is fixed and when to add a line, and sitemap() no longer carries a filter that never removed anything. CI is green. Thanks for the quick turnaround, and sorry for the wrong note about the description.
|
Owner check after deploy: |
…X-G6) (#478) Closes #448 Closes #456 New OG images: <img width="819" height="424" alt="Image" src="https://github.com/user-attachments/assets/bce3fb5e-d188-44d6-9ae9-e4a778685473" /> Method pages og images: <img width="849" height="436" alt="Image" src="https://github.com/user-attachments/assets/34374576-5bdb-44ea-85ee-a22d477a4596" /> Makes `northguild.dev/gmt/` easier for search engines and AI answer engines to crawl, index and quote. Everything follows documented mechanics: crawlable text, accurate sitemaps, one indexable URL per page, lighter pages, and an `llms.txt` that fits in an agent's context. AI crawlers stay allowed. No `packages/gmt` change and no changeset. ## Search descriptions - No two indexable pages share a meta description. A reference function's description is its TSDoc summary. Where sibling functions share one, a qualifier names the value kind or the namespace. - Each namespace has one written summary. It is both the namespace page's meta description and its line in `llms.txt`, and it replaces 13 identical boilerplate descriptions. - A test on the built site fails when two indexable pages share a description, and lists them. ## Crawling and `llms.txt` - Generated pages carry a real Last updated date from the newest commit of their inputs. The sitemap's `<lastmod>` is copied from that footer date, so the two agree. - A tracked `_headers` file caches hashed `/gmt/_astro/*` files for a year as immutable, and marks the `.md` twins and `llms*.txt` files `noindex`. - The 404 page carries `noindex` and no canonical. - The root `/gmt/llms.txt` fits in 64 KiB (it was about 213 KB). Its Reference section links one `llms.txt` per namespace. Every page carries `rel="describedby"` pointing at its namespace file, or at the root file outside the reference. - Each page keeps its own namespace's tree in the sidebar, and every other namespace shows one collapsed Overview link. Page HTML is 33% to 47% smaller on the four pages measured, and previous and next links are unchanged. - `deploy-smoke.mjs` checks the crawl rows after each deploy. ## Share cards and tags - Every share card shows the Earth, with its text inside the centre square that link-preview apps crop to. Cards stay under 300 KB. No card text is cut with an ellipsis; a title that does not fit fails the build. - Share tags: `og:locale` is `en_US`, `og:image:type` is set, `og:image:alt` and `twitter:image:alt` describe the card, and both image URLs end in `?v=<hash of the PNG>` so a redrawn card is a new URL. - A test on the built site checks every indexable page for exactly one of each tag, equal image URLs and a hash that matches the PNG. - Upload-ready copies of the cards are in `apps/dox/assets/share/`. ## Changelog pages - The changelog is one page per release line (`/changelog/1-21/`) plus an index, so no page nears 1 MB. Apple's Messages preview reads only the first 1 MB of a page, and the single page was 1,546,542 bytes. The largest page is now 729,975 bytes, and a build test fails on any page over 1,048,576 bytes. ## Root site card (other repo) The landing page `northguild.dev/` is a separate site. Its search data and card are made in `northguild/northguild.dev`, and both PRs are merged: - northguild/northguild.dev#21: sitemap, logo and Organization data. - northguild/northguild.dev#22: a card that survives a square crop, under 300 KB, with a versioned image URL. ## Globe, LCP and INP - The home globe starts when the page is idle and sets up in slices that yield to the main thread, so no task runs long. - Every globe pauses while off screen, while the tab is hidden, and while the search dialog or the mobile menu covers it. - Where the globe falls back to canvas 2D, it draws in a module Web Worker on an `OffscreenCanvas`. Input no longer waits for a globe frame. A probe compares text rasterisation on both paths first; if they differ, or the worker fails, the main-thread renderer takes over. The look is unchanged. - The hero clock loads `@northguild/gmt` by dynamic import beside the star list, which keeps the Temporal polyfill off the first-paint path. - The clock list measures its rows in one pass after writing them. - `apps/dox/scripts/web-vitals.mjs` measures LCP, CLS, TBT and INP by hand. Local lab numbers for the home page on mobile, before then after: TBT 464 ms to 149 ms; LCP 3.0 s (varying between runs) to 2.26 s; INP at 4x CPU throttling 256 ms to 96 ms with WebGPU, and 392 ms to 96 ms with canvas 2D. Guide and reference pages were already good. ## Owner steps From `context/domination/issues/CORE-86/search-and-crawlers.md` (runbook §8). - [x] Rich Results Test on `https://northguild.dev/` (northguild/northguild.dev#21): Organization, 0 errors, 0 warnings. - [ ] **O1** Search Console: submit both sitemaps and record the Pages report. - [ ] **O2** Search Console URL Inspection on `/gmt/` and two reference pages. - [ ] **O3** Bing Webmaster Tools: import from Search Console, then submit the sitemaps. - [ ] **O4** AI Crawl Control: record the 30-day request counts per crawler. - [ ] **O5** Crawler Hints: turn it on. Turn it off if Bing lists no IndexNow URLs within a week. ## Open checks after deploy These run against the live site, so they stay open. The list is also in `context/dox/built.md`. - [ ] Facebook Sharing Debugger, LinkedIn Post Inspector, X, Slack, Discord, WhatsApp: the card shows for `/gmt/`, a guide, a reference page and `/gmt/changelog/`; WhatsApp shows the whole title in a square crop. - [ ] iMessage: the rich preview shows for the same URLs, including a changelog line page. - [ ] The same platform checks for `https://northguild.dev/`. - [ ] `node apps/dox/scripts/deploy-smoke.mjs` passes against `https://northguild.dev`. - [ ] Rich Results Test and validator.schema.org read the Organization data on `https://northguild.dev/`. - [ ] `node scripts/web-vitals.mjs --live https://northguild.dev` with `PSI_API_KEY` set records PageSpeed Insights and CrUX numbers. ## Baseline Search Console, Bing and AI Crawl Control numbers from before deploy: _To be filled in by the owner._ ## Validation - Each slice was checked by a tester against its definition of done: - Search, crawl and `llms.txt`: the Dox Vitest suite, every built-site test on a fresh build, `astro check`, lint, a local Wrangler run of `deploy-smoke.mjs`, and a whole-site HTML comparison against `main`'s build. - Lean sidebar: the browser and keyboard checks, and the HTML sizes listed above. - Share cards and tags: card unit tests (text inside the square, contrast at least 4.5:1, no ellipsis) and `share-preview.test.ts` on a fresh build. - Changelog pages: the size test on a fresh build. - Globe: pixel comparison of the worker against the main-thread renderer, and the pause rules, in Chromium. Interior pixels differ by 1 to 14 levels where the ocean pattern is filled, and the clock card can show one frame before the globe when it mounts. Both are accepted. - Every test that reads the built site now runs in CI, found by rule (`apps/dox/scripts/dist-tests.mjs`). All 11 files pass on a fresh build. - CI on 9a3fee6: every job passed, including the Dox gates. The review fixes are in d516fc3. - Local Lighthouse, mobile, home page: performance 78 to 84 on `main`, 93 to 95 here. - `pnpm deps:sync`: no change to any `Blocked by` cell. Not verified: - `verdict validate` has not run locally: the scratch copy it would run in is not a git checkout. CI runs the same build, lint and test steps. - The globe worker has not run in real Firefox or Safari. Firefox would not start on the test machine.
Part of northguild/gmt#448 (DOX-G6). The
/gmt/docs changes for that issue are in their own PR in the gmt repo; this PR covers the landing page only.What changes
@astrojs/sitemap3.7.4 (pinned) writessitemap-index.xmlandsitemap-0.xml, which listshttps://northguild.dev/. The package leaves the 404 page out itself. Nolastmod, since the site has no per-page date that is verifiably accurate.Sitemap: https://northguild.dev/sitemap-index.xml, goes before the/gmt/one, andDisallow: /gmt/api/keeps crawlers off the docs chat API (JSON endpoints only, no pages). Every other line is unchanged,Content-Signal: search=yes, ai-input=yes, ai-train=yesincluded.public/logo.png, 512×512, rendered fromfavicon.svg. To re-run:rsvg-convert -w 512 -h 512 -f png -o public/logo.png public/favicon.svg(librsvg 2.63.2), fromapps/site. The mark keeps its own dark tile, so it shows on white.OrganizationJSON-LD block:name,url,logoandsameAs(https://github.com/northguild).<is escaped as\u003c, so a value cannot close the script tag.noindex, and no canonical link. Before,/nopeanswered 404 with a canonical pointing at/404/, which answered 200.aurora.worker.ts.setTimeoutbecomesself.setTimeout, as the file already does elsewhere. The sitemap package brings in@types/node, which typed a baresetTimeoutas Node'sTimeoutand failedpnpm check. No change in the browser. The same dependency changes the astro, vite and sharp keys inpnpm-lock.yaml.Checks
pnpm lint,pnpm format:check,pnpm checkandpnpm buildpass.dist/: the sitemap lists/and no 404;robots.txthas bothSitemap:lines;logo.pngis a 512×512 PNG;index.htmlhas exactly one JSON-LD block,@typeOrganization;404.htmlhasnoindex, no canonical and no JSON-LD.wrangler dev:/sitemap-index.xmland/logo.pnganswer 200;/nopeanswers 404 withnoindex./at 1440 and 390 looks the same before and after. The change is in<head>only.Owner steps after deploy
https://northguild.dev/sitemap-index.xmlin Google Search Console and Bing Webmaster Tools.https://northguild.dev/in Google's Rich Results Test and at validator.schema.org. Both should read the Organization with no errors.