ai-crawlers

Answer Engine Optimization for the modern web. Make your site discoverable by ChatGPT, Claude, Perplexity & AI search engines. Generates llms.txt, robots.txt, sitemap, JSON-LD & more.

Self-hosted AI crawler monitoring from Nginx / OpenResty access logs. FastAPI + Vue, source verification, page coverage and collector health.

Automate web workflows on real browser tabs without hosting a browser farm.

Self-hosted Google Analytics alternative that reads your server access log: real people, AI assistants, crawlers, bots and the scanners probing your site, which JavaScript analytics never see. Leads and revenue by source. Every figure counted twice. No cookies, no third-party script. Python, SQLite, Docker.

Curated GEO platforms, repositories, and verification scripts for AI-era discoverability.

Automatically generate llms.txt and llms-full.txt files for Laravel applications to help AI models and LLM crawlers understand your website.

One Cloudflare WAF custom rule for WordPress: blocks bad bots, AI training crawlers, spoofed Googlebots, and common WordPress attack paths. Fits the Free plan.

ScrapeLock – Content & AI Bot Protection for WordPress: blocks scrapers, AI training bots and fake Googlebots, rate limits, token-protected XML feeds.

Moteur open source d'analyse de logs SEO / GEO : qui crawle vraiment votre site (moteurs, bots IA, usurpateurs), crawl budget, visibilité IA, bilan avant/après — report.json prêt pour votre propre surcouche. Atelier Teknseo 2026.

Scope, audit, harden, and retest web projects with AI coding agents and reproducible evidence.

Classify web traffic from server log entries and header capture into human, AI agent, AI crawler, AI assistant, programmatic, and other bot categories. No analytics service or middleware required.

Is your site ready for AI search? Open-source SEO, AEO & GEO audits: AI crawler access, llms.txt, answer-ready pages, AI visibility, and an AI analyst's to-do list. CLI, GitHub Action, Claude Code plugin, MCP server.

Check whether AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more) can access your site. CLI, library, MCP server and GitHub Action.

🤖 Generate optimized robots.txt files for AI search engine crawlers (GPTBot, PerplexityBot, ClaudeBot, and more)

Check whether a website is reachable and readable by AI crawlers, and score it. Two scores, never averaged. Blocking model training costs no points.

PHP library to block unwanted crawlers, scrapers and bad bots by User-Agent, IP block lists and allow lists. No captcha, works on any PHP site. Self hosted alternative to Cloudflare.

Curated list of AI visibility / GEO / AEO resources: tools, AI crawler datasets, agent skills, llms.txt, MCP servers, research papers, and guides.

Laravel SEO package for metadata, canonicals, Open Graph, linked JSON-LD, sitemaps and crawler controls — free and MIT licensed.

Make your website work for agents - audit and fix how AI crawlers, answer engines, and agents discover, read, cite, and act on a site. Live probe + codebase fixes, scored across 6 pillars.

Server-side AI crawler & bot traffic tracking for Laravel — reports ChatGPT, Claude, Googlebot, GPTBot & co. to DataFast. Unofficial PHP port of @datafast/ai-crawl.