We crawled 190 SEO sites: what they tell AI bots and what their servers do

Sites that write about search are the ones the rest of the web copies. So we pointed GLLmolfar at 190 of them on 4 November 2026 and read what they actually serve: news outlets such as Search Engine Roundtable and PPC Land, the blogs of Ahrefs, Semrush, Moz and 62 more tool makers, experts, agencies, Google's and Microsoft's own search blogs, and SEO sites in five other languages.

For each site we read robots.txt for the 14 AI crawlers the app tracks, fetched llms.txt, requested the start page as a browser and as four AI crawlers, and crawled up to 50 pages from the news or blog section. 156 sites answered with HTML; that gave 6,442 pages, 1,643 of them articles from 111 sites.

1. Five days in, nobody uses the shared training token

On 30 October Google and Microsoft backed a shared robots.txt token that opts a site out of AI training across providers: a group for User-Agent: __usage_ai-training. On 4 November not one of the 157 robots.txt files we could read contained it. None of the home pages carried Bing's new ai-training or ai-grounding values in a robots meta tag either.

That is not a verdict on the token — Google says it aims to support it in 2027 — but it is a useful baseline. If you add it now, you are ahead of every SEO publication we looked at.

Twelve sites already use a different mechanism: a Content-Signal line, the format Cloudflare promotes. Five of them carry Cloudflare's managed text word for word (search=yes, ai-train=no, use=reference). Three say the opposite — ai-train=yes, ai-input=yes — and invite AI use explicitly. Three allow AI answers but not training, and one refuses both.

2. Most AI blocks are someone else's default

Only 13 of 157 sites block any AI crawler in robots.txt. The blocks are concentrated in one group: 7 of 22 marketing and media outlets block GPTBot, against 1 of 57 tool vendors. Five of the 13 blocking files contain the line # BEGIN Cloudflare Managed content and the same list of eight training crawlers — GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent and Amazonbot. Those sites did not write a policy; they turned one on.

Every one of the 11 sites that block GPTBot leaves OAI-SearchBot open. The split between training and answering, which took a year of argument, is now the default shape: only one site in the sample, WebmasterWorld, blocks an AI search crawler at all.

The lists are also old. 15 robots.txt files name anthropic-ai or Claude-Web, tokens that Anthropic's current crawler documentation does not list; 13 of them name ClaudeBot as well. Only 5 name Claude-SearchBot and 6 name Claude-User, the two agents that fetch pages for Claude's answers. A robots.txt that was copied once and never revisited says no to crawlers that no longer come and nothing to the ones that do.

3. robots.txt says yes, the server says no

We then asked each start page for its HTML four more times, with the user agents of GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot. Of the 147 sites that answered a browser, 19 (13%) refused at least one AI crawler that their own robots.txt allows: a 403, a 429 "too many requests" on the first request, or a Cloudflare block page. ClaudeBot was refused most often.

Read this with one caution. Our requests carried the crawlers' names but came from our own address, not from the crawlers' published IP ranges, and some firewalls block exactly that: a request that claims to be GPTBot from somewhere GPTBot never crawls from. So a refusal here does not prove the real crawler is refused. What it does prove is that robots.txt is not the whole policy. The rules that decide whether an AI crawler gets a page often sit in a firewall or a CDN that a robots.txt audit never sees, and the only place to check them is your server logs.

4. Some SEO sites cannot be audited by an SEO crawler

Eleven sites (6%) answered every request that did not come from a real browser with a Cloudflare challenge — our crawler's own user agent and a browser user agent from a plain HTTP client alike. They include Search Engine Land, Search Engine Watch, Neil Patel, WordStream, The Drum and Marketing Week. Two more, Marketing Dive and Social Media Today, served the browser user agent and challenged the crawler's.

Three of the eleven publish an llms.txt for AI systems, and one of them says ai-train=yes, ai-input=yes in its robots.txt. The invitation is written in one layer and withdrawn in the next. Verified crawlers from Google, Bing and OpenAI may pass the check; a desktop SEO crawler, a monitoring script or a smaller AI service will not.

5. One in three has an llms.txt, and they are not alike

59 of 190 sites (31%) serve an llms.txt. Agencies lead with 11 of 25 (44%), then tool vendors with 26 of 65 (40%); 4 of 29 media outlets have one, and among the official search blogs only Microsoft Advertising does.

The files disagree about what llms.txt is for. The median file lists 65 links, the smallest 4. Six list more than 500, and one nearly 5,000 — a sitemap with a different name, which defeats a format meant to point a model at the few pages that explain a site. Two sites publish an llms.txt and block the training crawlers in robots.txt, which is the consistent version of the policy: answer from my pages, do not train on them.

6. The articles are well marked up — except for Discover

On the 1,643 articles the crawl found, the trust signals are nearly universal: 94% carry an Article-family schema type, 95% name an author, 88% describe the author as a Person, and 91% state a modified date. The median article is 1,978 words with 7 H2 headings and 8 links out to sources; 57% have at least one heading phrased as a question.

The gap is in the robots meta tag. A third of the articles (33%) do not carry max-image-preview:large, the setting that lets Google Discover show a large image. Among news outlets, 126 of 196 articles are typed NewsArticle; the rest use the generic Article.

7. The basics, on 6,442 pages

Across every HTML page the crawl read — articles, category pages, landing pages — the familiar faults are still there, and the ones that matter for assistive technology are the most common.

The JSON-LD errors cluster in three places: breadcrumb items without a position (180 pages), JSON-LD that does not parse at all (115 pages), and SoftwareApplication markup on tool pages without a rating or review (102 pages), which makes it ineligible for the rich result it was added for.

The pages are heavy but fast. The median HTML document is 178 KB before any script or image, and one in ten is over 554 KB; the median server answered in 300 ms, one in ten took over 1.3 seconds.

What to do with this

1. Decide training and answers separately, per crawler. Write the decision in robots.txt, and if you want the shared token, add the __usage_ai-training group now. 2. Check that the server agrees with robots.txt. Look for GPTBot, ClaudeBot, Claude-SearchBot, OAI-SearchBot and PerplexityBot in your logs, and for 403 and 429 answers to them in your CDN or firewall. 3. Prune the old tokens. Replace anthropic-ai and Claude-Web with the agents that exist today. 4. If you turned on a managed robots.txt, read it. It is a policy with your name on it. 5. Keep llms.txt short. A handful of links to the pages that explain what you do, not your sitemap. 6. Add `max-image-preview:large` to articles if you want Discover traffic. 7. Give every link and button a name. It is the most common fault in this sample and the cheapest to fix.

How this was measured, and its limits

The list of 190 sites is ours: news and trade outlets, SEO tool blogs, experts, agencies, the official search blogs, and SEO sites in German, French, Spanish, Polish and Ukrainian. 34 sites gave no HTML (a challenge, a 404 at the address we used, or no answer) and are left out of the page figures. The crawl took up to 50 pages a site from its news or blog section, so the page figures describe those sections, not whole sites. JavaScript was not run; word counts are of the HTML as served. robots.txt was read for 14 AI crawlers; the AI user-agent requests were made once, from one address.

Everything here can be repeated in GLLmolfar: the AI readiness view reads robots.txt for each AI crawler and checks llms.txt, and the Issues view lists the page checks above for any site you crawl.