AI search readiness: which bots can read your site and what they could quote

Steps in Molfar

1. 01

   Crawl the site and open the AI screen

   After a crawl, the AI screen shows a readiness score in four parts. Access and the text bots get are measured: robots.txt, sample pages fetched as each bot, text only after JavaScript, and nosnippet. Meaning and trust are the app's own estimate. The score never goes above the weaker of the two measured parts.

   AI — docs

   

2. 02

   See who robots.txt lets in

   The robots.txt, bot by folder table marks every bot in every folder with ✓ or ✕, and hovering shows the rule and the line that matched. Closing a training bot is a legitimate choice: Google's AI Overviews and AI Mode use Googlebot, not Google-Extended.

   AI · robots.txt by folder — docs

   

3. 03

   Check what the bots actually get

   Check as AI bots fetches about ten pages as a browser and with the User-Agent of every bot robots.txt allows, from your computer, without JavaScript. You see the status and the word count for the browser and for each bot. A CDN that verifies a bot's IP address may answer the real bot differently.

   AI · Check as AI bots — docs

   

4. 04

   Find what blocks quoting

   Can AI read and quote it lists the problems with the number of pages each touches: measured ones first, such as nosnippet or max-snippet: 0, then style checks marked heuristic. Heuristics do not move the score, and each line opens its pages.

   AI · Can AI read and quote it — docs

   

5. 05

   Check which questions the site answers

   Paste the questions your customers ask, one per line, and press Match questions. The embedding model on your computer compares each one with every section of the site and shows the closest. With Search Console connected, you also get queries phrased as questions, with impressions and clicks.

   AI · Which questions the site answers — docs

   

6. 06

   Check llms.txt and Markdown for agents

   Molfar shows whether llms.txt and llms-full.txt exist, their sections and any links that lead to errors. For the sample pages, it shows whether the site answers Accept: text/markdown with Markdown and whether a .md version exists. No major AI search engine has said it reads llms.txt, and the app says so.

   AI · llms.txt — docs

   

7. 07

   Save the reports

   The AI readiness and Page citability reports each answer one question: can bots fetch and read the site, and how might a retriever cut a page into passages. Both save as HTML or PDF.

   Reports — docs

   

What you get

Good to know

• The readiness score is Molfar's own estimate, not a standard; only access and the text bots get are measured.
• The check runs from your computer: a CDN that verifies the real bot's IP address may answer it differently.
• Matching questions needs embeddings, which are built on your computer during the crawl.

Questions

If I block GPTBot, will my site disappear from ChatGPT?

Blocking a training bot is a separate choice. Bots that fetch pages to answer questions have other names, and Molfar shows each one separately.

Does a high score mean I will be quoted?

No. The score shows whether bots can read the site; what goes into an answer is up to the assistant.

Do I need llms.txt?

Molfar checks it if you have one, but no major AI search engine has said it reads it.