Free Open source MIT
Vectorscope

A Claude Code skill for AI search: read your page the way a retriever does

The Vectorscope skill is a free Claude Code skill for AI search that fetches one public page, keeps the text a crawler keeps, cuts it into the passages a retriever stores, and shows what falls past the 26-passage cap — with no model or embedding calls of its own.

Get it on GitHub How to install Try the web Vectorscope

Updated 16 October 2026Sources checked 16 October 2026MIT licenceVersion 2.103

01 What happens to a page before an AI assistant can quote it

Before an AI assistant can quote a page, a machine has to fetch it, reduce it to text, cut the text into passages and store some of them; the assistant then retrieves passages, not pages. That is the shape of retrieval-augmented generation as it was named in 2020: a retriever finds passages in an index and a generator writes from them [16]. The original paper split Wikipedia into 100-word passages before indexing anything [16].

The fetch is done by named crawlers that publish what they are for. OpenAI separates the bot that gathers training data from the one that finds pages for ChatGPT search and the one that fetches a page a user asked about [9]; Anthropic and Perplexity make the same distinctions for their own bots [10] [11]. Google says a page has to be indexed and eligible for a snippet to appear in its AI features [1], and Bing says Copilot rests on the same crawling and indexing as its search [14].

What the crawler keeps is text. Googlebot renders JavaScript in a later phase [7], but when Vercel measured its own network in December 2024, the major AI crawlers fetched JavaScript files without executing them [8], and Common Crawl says its CCBot does not execute JavaScript either [12]. Words that only exist after a script runs are, for those crawlers, words that do not exist.

  1. Fetch One GET of the HTML, as a crawler would. No scripts run.
  2. Text Menus, footers and markup removed; the readable text remains.
  3. Passages Cut at every h2, then into windows of about 2,000 characters.
  4. Cap At most 26 passages kept. Everything after is never stored.
  5. Retrieval A question is matched against passages, never against the page.

The cut and the 26-passage cap are the rules of our free Vectorscope and of this skill. No assistant publishes its own passage limit; 26 is a working model of a limit, not a number taken from any of them.

02 Why every passage has to stand alone

A passage that needs the passage before it to make sense cannot be quoted, because the retriever hands it over alone. Anthropic's write-up of contextual retrieval describes exactly this failure: once a document is split into chunks, an individual chunk can lose the context that said what it was about [18]. Pinecone's chunking guide states the working rule from the other side — if a chunk makes sense to a person without the text around it, it will make sense to the model too [19].

Leans on the passage before

It also works offline. Pricing for this starts at the second tier, and it can be switched off per project.

Stands alone

Does the editor work offline? Yes — the desktop editor works offline. Offline mode is included from the second tier and can be switched off per project.

An illustrative pair, written for this page. The first passage is true and useless on its own: "it" and "this" point at a heading the retriever did not send.

Position matters even after retrieval. Liu and colleagues found that language models often use information best when it sits at the beginning or the end of a long context, and markedly worse when it sits in the middle [17]. A page cannot choose where an assistant puts its passage; it can make sure the passage carries its own subject, so that wherever it lands it still says what it is.

Most pipelines cut by size first. The common default in RAG tooling is a fixed chunk size with an overlap, split at paragraph and line breaks where it can be [20]. A window cut that way does not know where your answer ends.

One honest caveat about Google: it says Google Search does not need content broken into small pieces, because its systems find the relevant part of a page on their own [2]. A Claude Code skill for AI search cannot change that, and this one does not try. It is about the retrievers that store passages, and Bing, for one, says a URL is more likely to be selected for grounding and citations when its content stands on its own [14].

03 What the skill does, step by step

A skill is a folder with a SKILL.md file of instructions that an agent loads. Claude Code finds skills in your personal and project skill folders, uses one when it is relevant, and lets you call it by name, here as /vectorscope [21]; other agents load the same folder from their own skill directories, listed further down. As a Claude Code skill for AI search, Vectorscope splits the work between two scripts that follow fixed rules and the steps that are the agent's judgement — made by the model your agent runs, which in Claude Code is Claude.

  1. Read the page

    dist/read.mjs fetches the URL with one plain GET and reads the site's robots.txt, refuses private and local addresses, extracts the visible text, cuts it at every h2 and then into windows of about 2,000 characters with 200 characters of overlap, applies the 26-passage cap and marks passages that look like link lists or small print. With --render it also loads the page in a local Chrome, Chromium or Edge with a throwaway profile and reports how many words exist only after JavaScript runs; the reading itself stays the plain fetch.

  2. Group the passages

    The agent reads every passage in full and writes groups.json: at most six groups by subject, each named in the page's own language. This is a judgement — the web tool groups by embeddings, the skill by reading — and the next script refuses a grouping that leaves a passage out or counts one twice.

  3. Write the report

    dist/lens.mjs validates the grouping, runs every check and writes a self-contained HTML report, Markdown and JSON; the report is always shown, not just named. Every section and number carries a tag — measured by the parser, derived by a fixed formula, or the agent's judgement — and a "?" beside each number opens its formula, unit, threshold and limits. The info panel at the top explains the tags and names the model that judged. EN | UK switch, two themes, and a content security policy that blocks every network request.

  4. Show where the text goes

    A character waterfall follows the page from the raw HTML to the passages: removed by the cleaner, kept, in passages, past the cap. A table lists what the cleaner removed and why — the element, its CSS path, its characters, how its text begins — and a section map shows how much of each h2 reached a passage. Past the cap is counted from the unique text in passages: 55,001 characters for the Wikipedia example, where subtracting every stored character, overlaps and intro included, says 52,973.

The report also lists the page's own vocabulary, phrases first: two to four words the page writes together in at least two passages, then single narrative words no listed phrase already used. It is what the page says, not search demand; for how people ask, use the query test.

The top of the skill's 2.102 report for the Wikipedia article on large language models: Measured, Derived and Agent's judgement tags, the info panel that explains them, and the counters with 55,001 characters past the cap.
The head of the report for a page that hits the cap (the skill's Wikipedia example).
  1. A tag beside every number: measured, derived or the agent's judgement.
  2. The info panel says what each tag means.
  3. 55,001 characters past the cap, counted from unique text.
  4. Unique text in passages, beside the total with overlaps and intro.
  5. The "?" opens how a number is computed.
A metric's explanation opened from its question mark: characters past the cap, with its formula, unit, threshold and known limits.
The "?" beside a number, open.
  1. The "?" beside the number.
  2. The formula, in words.
  3. Known limits, including what the old count did.
The character waterfall for the Wikipedia article: text in the raw HTML, removed by the cleaner, kept, in passages unique, kept in no passage, and past the cap.
Where the text goes, from the raw HTML to the passages.
  1. What the cleaner removed: scripts, styles, navigation.
  2. Unique text that reached a passage.
  3. Past the cap: kept, but in no passage.
  4. The passage total split into unique, overlap and intro.
The section map for the Wikipedia article: every h2 with how many of its characters reached a passage; the last six sections reach none.
How much of each section reached a passage.
  1. Sections that lost text, and sections that reached none.
  2. A section the cap cut in the middle.
  3. Sections past the cap: 0 characters stored.
What the cleaner removed from the Wikipedia article: reasons with characters and counts, then each element with its CSS path, characters and the start of its text.
What the cleaner removed, largest first.
  1. Removed characters by reason.
  2. The element and its CSS path.
  3. Its characters, then how its text begins.
The page's vocabulary in the skill's report: phrases this page owns, with uses, passages and groups, then narrative words, under the note that this is what the page says, not search demand.
The page's vocabulary for the web Vectorscope's own page.
  1. Derived: counted by a fixed rule, not judged.
  2. What the page says, not search demand.
  3. A phrase the page owns: uses, passages, groups reached.
  4. Narrative words: single words no phrase already used.
The 2.102 report for the web Vectorscope's page with its interface in Ukrainian: the tags and the info panel in Ukrainian, the page's own title in English.
The EN | UK switch changes the report's interface, never the page's text.
  1. The interface language switch.
  2. The tags and explanations, in Ukrainian.
  3. The page's own title is never translated.

04 The automatic checks, and why each one matters

After the reading, the Claude Code skill for AI search runs its checks with no judgement involved: each finding has a code, a severity, the passages it rests on and the evidence, and every threshold is a named constant in the skill's code. The report ranks them by severity, and crawler access comes first, because a blocked crawler makes the rest moot.

  1. Answering crawlers blocked in robots.txt ai_search_blocked

    The skill reads /robots.txt on the page's own origin and matches it the way RFC 9309 describes for twelve crawlers. If a search or user-fetch crawler — OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot or Bingbot — is disallowed, the report says so before anything else; Google's own advice for its AI features starts with the crawler not being blocked [5]. robots.txt is a request crawlers are asked to honour, not access control [13], and each operator documents its own agents [9] [10] [11]; fetchers acting for one user are a separate case, since OpenAI says robots.txt rules may not apply to ChatGPT-User and Perplexity says Perplexity-User generally ignores them. A robots.txt that answers 5xx or not at all is reported as robots_unreachable.

  2. Only training crawlers blocked ai_training_blocked

    GPTBot, ClaudeBot, Google-Extended or CCBot disallowed is usually deliberate and does not stop an assistant citing the page today, so it is a low finding. Google says Google-Extended does not affect inclusion in Google Search [4], and robots.txt controls crawling rather than whether a page appears in results [3]; Common Crawl's CCBot reads robots.txt first [12].

  3. The page asks not to be indexed noindex

    A meta robots noindex means the search indexes assistants answer from will drop the page. Google's AI features show only pages that are indexed and eligible for a snippet [1].

  4. Text that exists only after JavaScript js_dependent

    With --render, a page where JavaScript adds 50% more words or more (and at least 150) is flagged. When Vercel measured its network in December 2024, the major AI crawlers fetched JavaScript files without executing them [8], so they read the smaller number.

  5. Text past the 26-passage cap cap_overflow

    All 26 slots are filled and text is left over. Whatever comes after the 26th passage is never stored by this pipeline, so no question can bring it back. The cap is ours, not any assistant's.

  6. FAQ answers that lost their questions lost_faq_questions

    Three or more sentences open with "Yes." or "No." while the page has fewer question marks than half of them: the questions sat in buttons or accordions that did not survive extraction, and each answer has lost its address. Google retired FAQ rich results in May 2026 [6], so the markup was never the point — the text is.

  7. One heading holding over 30% of the text dominant_heading

    The chunker splits only at h2, so a section that large is cut by length alone; its subheadings are h3 or styled text, and neither makes a cut.

  8. Passages that open mid-sentence mid_sentence_start

    A passage that starts in lowercase or on punctuation was cut by the 2,000-character window. It opens without its subject — the failure described in the stand-alone section [18].

  9. Navigation inside the content nav_in_content

    Four or more consecutive lines of one or two words inside the main text: menus, tag lists or label tables became passage text and spent passages. Google asks for important content to be available as text [1]; chrome that reads as text is spent as content.

  10. Thin sections thin_section

    A heading with fewer than 150 characters under it is too short to answer anything on its own.

  11. Truncated teasers truncated_teaser

    Lines ending in "…" or "..." (code samples excluded) hand a retriever half a claim; three or more raise the severity to medium.

  12. The same claim in every section repeated_claim

    The same four-word phrase — with grammar in it, not code — in at least half the groups. Repetition spends passages on what was already said.

Then the markup: a canonical that is missing, relative, duplicated or pointing at another URL; JSON-LD that does not parse or is absent; and a meta description that is missing, under 70 or over 160 characters, because the chunker puts the description into passage #0.

robots.txt first: which AI crawlers may read the page

The table is the robots.txt verdict from a real run of the skill on the web Vectorscope's own page: twelve crawlers, each with the rule that decided it. On this page every one of them is allowed.

CrawlerVendorWhat it is forThis pageDeciding rule
OAI-SearchBotOpenAIsearch answers allowedAllow: /vectorscope
ChatGPT-UserOpenAIa fetch a user asked for the vendor says robots.txt may not apply to it allowedAllow: /vectorscope
GPTBotOpenAItraining data allowedAllow: /vectorscope
Claude-SearchBotAnthropicsearch answers allowedAllow: /vectorscope
Claude-UserAnthropica fetch a user asked for allowedAllow: /vectorscope
ClaudeBotAnthropictraining data allowedAllow: /vectorscope
PerplexityBotPerplexitysearch answers allowedAllow: /vectorscope
Perplexity-UserPerplexitya fetch a user asked for the vendor says robots.txt may not apply to it allowedAllow: /vectorscope
Google-ExtendedGoogleGemini training and grounding allowedAllow: /vectorscope
GooglebotGoogleweb search index assistants answer from allowedAllow: /vectorscope
BingbotMicrosoftweb search index assistants answer from allowedAllow: /vectorscope
CCBotCommon Crawlopen web corpus allowedAllow: /vectorscope
Source: the skill's example run examples/judgement/getlooploop-vectorscope/reading.json · https://www.getlooploop.com/robots.txt · https://www.getlooploop.com/vectorscope · 16 October 2026 · skill 2.103 (bbb7afa)
The AI-crawler table in the skill's report: twelve crawlers with vendor, purpose, the verdict for this page and the deciding robots.txt rule.
The same verdict as it appears in the report, before any other finding.
  1. User-fetch crawlers carry the vendor's own caveat about robots.txt.
  2. The verdict for this page's path.
  3. The rule that decided it, and the group it came from.
  4. Where the file was read, and the matching rules used (RFC 9309).

05 From a reading to a rewrite: query test, plan, compare, before and after

A reading says what is wrong; the rest of the Claude Code skill for AI search helps fix it and shows whether the fix worked. The query test and the plan are the agent's judgement, and the script checks both: a quote must be copied verbatim from the passage it cites, or the run is refused with the item named.

A query test

The agent writes 5 to 10 typical queries — yours if you give them, otherwise the questions a person would type into an assistant, including at least one the page probably does not answer. For each it names the single passage a retriever should return, a verdict (complete, partial or missing), whether the words make sense quoted alone, and the exact words that answer.

A rewrite plan

Fixes in priority order, each tied to a check code and to the passages it rests on, with before → after rewrites for the weakest passages. The order follows what changes retrieval most: blocked crawlers and noindex, then text past the cap, then passages whose questions or subjects were lost, then thin or fragmented sections, then markup. A rewrite may not add a claim the page does not make.

Compare mode

Read 2 to 5 pages in one run. The agent writes one shared topic dictionary first, so the groups line up across pages; the report shows a topic × page table, topic gaps (topics the others cover and a page does not), fact gaps (facts only one page states) and who wins each query. The skill compares the pages you name and is told not to volunteer a comparison against a named company's page.

Before and after

Keep the old reading. After the page changes, read it again and pass --baseline old-reading.json to the report, with --baseline-groups if you kept the old grouping: it shows which passages are unchanged, changed, added and removed, the cap status before and now, and which old groups the new ones came from. After a real change, IndexNow can tell participating search engines the URL changed [15].

The rewrite plan in the skill's report: numbered items, each with the check it answers, the passages, the problem, the fix and a before and after rewrite.
The rewrite plan for the web Vectorscope's own page.
  1. Tagged as the agent's judgement.
  2. The check the item answers, and its passages.
  3. Before: the passage text, copied verbatim.
  4. After: the rewrite, which names its subject so it stands alone.
The query test in the skill's report: eight queries with the best passage, a complete or partial verdict, whether it is citable, and the words that answer.
The query test: eight typical queries against one page.
  1. Tagged as the agent's judgement.
  2. The summary: answered completely, partly, not at all, and citable.
  3. The verdict: complete, partial or missing.
  4. The exact words that answer, checked against the passage.
Compare mode in the skill's report for MDN's HTTP caching guide and web.dev's HTTP cache article: losses by page, a topics × pages table, and the topic gaps of each page.
Compare mode: MDN's HTTP caching guide against web.dev's HTTP cache article.
  1. Shown in 2.102: MDN lost 167 characters past the cap. 2.103 fixed the cut behind it (an h3 that repeated an h2 split a section), and the page now fits.
  2. Passages and characters each page spends on a topic.
  3. A topic one page covers and the other does not.
  4. The topic gaps, page by page.

06 Two real pages, read by the skill

Two runs ship with this Claude Code skill for AI search as examples. The Wikipedia article on large language models fills all 26 slots and leaves 55,001 characters past the cap, counted from the unique text in its passages; 2.101 subtracted the overlaps too and said 52,973. The Vectorscope page needs 17 passages and loses nothing. Every figure below is read from those two runs' JSON at skill 2.102 — nothing is estimated.

Where the 26 slots go
  • passage
  • unused slot
Wikipedia — Large language model 26 / 26
1234565
getlooploop.com/vectorscope 17 / 26
123435654

Twenty-six slots, one per stored passage, in reading order; a gap marks a change of group, and an outlined slot is one the page did not need. The cells count passages, not characters — the characters are in the last chart.

Characters in each passage — Wikipedia, Large language model
05001,0001,5002,0002,000-character window1611162126

Long sections are cut into windows that stop near 2,000 characters; short bars are the ends of sections and the page's opening lines.

Characters in passages, counted once, and characters past the cap
  • in passages
  • past the cap
Wikipedia — Large language model39,11155,001 past the capgetlooploop.com/vectorscope13,5470 past the cap

Passage text is counted once, without the 200-character overlaps and the intro, so the two bars add up to the page's kept text, less the heading lines that sit in no passage.

Source: the skill's example runs examples/wikipedia-llm.json and examples/getlooploop-vectorscope.json, skill 2.102, generated 16 October 2026.

The figures as a table
PageWordsPassagesCharacters stored, with overlapsOf them uniqueCharacters past the cap
Wikipedia — Large language model13,71826 / 2641,85039,11155,001
getlooploop.com/vectorscope2,46317 / 2614,21113,5470

07 Works with your agent: Claude Code, claude.ai, Codex, opencode, Hermes

This Claude Code skill for AI search is one folder — a SKILL.md and Node scripts — and every agent runs the same commands from it, so the agent's own model does the reading. Claude Code is the agent it was built in and the one this page is named for, but nothing in the scripts depends on it. This page describes release 2.103 (its screenshots are from 2.102); versions count in thousandths (2.101, 2.102, 2.103 …), so the first number moves only on a breaking change. Each agent finds skills in its own place:

AgentWhere it looks for skills
claude.ai, Claude desktopvectorscope.zip, uploaded in Customize > Skills [26]
Claude Code~/.claude/skills/ (personal) · .claude/skills/ (project) [21]
OpenAI Codex~/.agents/skills/ (user) · .agents/skills/ (repository) [22]
opencode~/.config/opencode/skills/ · .opencode/skills/, and the Claude Code and Codex folders too [23]
Hermes Agent~/.hermes/skills/, plus project folders once you trust the repository [24]
Any other agentpoint its instructions file, such as AGENTS.md, at skills/vectorscope/SKILL.md

GLM is a model family, not an agent: Z.ai's GLM Coding Plan runs inside coding agents, and its documentation names Claude Code, Codex and OpenCode among them [25]. Install the skill for the agent you run GLM in, and GLM does the reading.

The full install commands for every agent are in the skill's README

08 How to install it: claude.ai, the desktop app, Claude Code

This Claude Code skill for AI search needs Node 20.18 or later and no npm install anywhere: its scripts ship bundled in dist/, so plain node runs them, even where package installs are blocked.

  1. claude.ai or the Claude desktop app: upload the zip

    Download vectorscope.zip from the release and keep it zipped. Turn on code execution, which skills need [26], then Customize > Skills > + > Create skill > Upload a skill.

    vectorscope.zip · 2.103

  2. Let the sandbox reach the page

    On Free, Pro and Max turn on Allow network egress; on Team and Enterprise an owner allows package managers and specific domains, with the site's domain listed [27]. If the setting cannot allow it, the run stops with REFUSED unreachable: use Claude Code.

  3. A pasted link is not an install

    Claude will not download and run code from a link pasted into a chat, because fetched content may carry instructions nobody vetted [28]; at most it imitates the steps, and those numbers are not the tool's.

  4. Claude Code and other agents: clone

    Anywhere on your machine.

    shell
    git clone https://github.com/Olexan-art/getlooploop-skills.git

  5. Copy the skill into your skills folder

    ~/.claude/skills/ makes it yours in every project; .claude/skills/ inside a repository makes it that project's.

    macOS, Linux, Git Bash
    mkdir -p ~/.claude/skills && cp -r getlooploop-skills/skills/vectorscope ~/.claude/skills/vectorscope

  6. The same copy in PowerShell

    On Windows without Git Bash.

    PowerShell
    New-Item -ItemType Directory -Force "$HOME\.claude\skills" | Out-Null
Copy-Item -Recurse getlooploop-skills\skills\vectorscope "$HOME\.claude\skills\vectorscope"

  7. Ask your agent

    Type /vectorscope https://example.com/page in Claude Code, or ask in plain words in any agent. The run ends with the HTML report shown: a file to download in claude.ai, opened or published in Claude Code.

09 What the skill does not do

A Claude Code skill for AI search can only show what a retriever receives. Everything past that would be a claim it cannot check, so it does not make one.

  • No score. A page is not given a grade, a rank or a percentage of "AI readiness".
  • No AI detection. It does not say whether a person or a model wrote the page.
  • No prediction of answers. It cannot say what any assistant will say about you, or whether any assistant has the page indexed today.
  • No stand-in for an assistant's own pipeline. The groups, the query verdicts and the plan are the agent's reading, not an embedding clustering, and the report's info panel names the model that judged them; two models can judge a page slightly differently. The fetch, the cut, the cap and the checks are fixed rules and do not vary.
  • No numbers made by hand. Where the scripts cannot run — no Node, no code execution, the network closed — the agent says so and stops rather than counting the page itself.
  • One public URL per reading, up to five in compare mode, read as an anonymous visitor: no login, no cookies, no Accept-Language. JavaScript runs only with --render, and the reading itself stays the plain fetch.
The foot of the skill's report: the note that the scripts fetched only the page and its robots.txt and called no model or embedding, and links to GetLoopLoop, X, Bluesky and the author's LinkedIn.
Every report ends by saying what it is not, and what was sent where.
  1. The scripts fetched the page and its robots.txt, sent nothing anywhere and called no model.
  2. Links to GetLoopLoop and its author.

10 How it relates to the free web Vectorscope

The web Vectorscope and the Claude Code skill for AI search share the fetch, the extraction, the cut and the cap. They differ in who reads the passages: the web tool embeds every passage, groups them by measured similarity and draws the map; the skill hands the passages to your agent and makes no embedding calls at all, which is also what lets it judge queries and write a plan.

The skill, in your agentWeb Vectorscope
Fetch, extraction, cut, 26-passage capSame rulesSame rules
GroupingThe agent reads and groupsEmbeddings, average-link clustering
Semantic mapNoYes
AI-crawler robots.txt table and automatic checksYesNo
Query test and rewrite planYes, by the agentOne question at a time, by similarity
Compare pages2 to 5 pagesOne URL at a time
Where it runsYour agent: your machine, or claude.ai's sandboxOur server, in the browser
LimitNone of oursA few runs per visitor

Open the web Vectorscope

11 Questions about the skill

Is this Claude Code skill for AI search free?

Yes. The skill is open source under the MIT licence. Its scripts make no model or embedding calls; the reading is done by the model your agent already runs, inside your own session.

Who is the Vectorscope skill for?

For people who write pages and for people who build what reads them. Content and SEO teams use it to check, before publishing, that the answer a page exists to give survives into one passage above the cap and that AI crawlers may fetch it. Engineers building RAG pipelines use it as a reference reading to hold their own chunker against.

Does the skill send my page anywhere?

No. The scripts make two GET requests — the page and its robots.txt — and send nothing anywhere; with --render, a local browser also loads the page. The report itself blocks every network request.

Is 26 passages the limit ChatGPT, Claude or Perplexity uses?

No. Twenty-six is the cap our Vectorscope and this skill apply. No assistant publishes a passage limit, and the number is a working model of a limit, not a measurement of anyone's.

Why does a page that looks long show only a few passages?

Usually because its text is drawn by JavaScript. The plain reading, like the major AI crawlers, does not run scripts; run it with --render to measure how much of the page exists only after JavaScript.

Can it tell me whether my page appears in AI answers?

No. It shows what a retriever would receive from a page. Whether an assistant cites you is a measurement over time on prompts you choose, which a reading of a page cannot answer.

Does it work outside Claude Code?

Yes. claude.ai and the Claude desktop app take it as an uploaded zip and run it in the code-execution sandbox; Codex, opencode and Hermes Agent load it from their skill folders; GLM runs it through those agents. The scripts are plain Node with nothing to install; the judgement is the agent's model.

12 Sources

How we checked these sources

This page explains a Claude Code skill for AI search, and every claim on it about how a search engine, a crawler or a retriever behaves is tied to a source, and each source was opened and read on 16 October 2026. Primary documentation comes first — the search engines and crawler operators describing their own systems — then the standard, then peer-reviewed research, then vendor writing on chunking where it states a working rule.

For each one we kept the exact sentence the claim rests on, in the source's own words, and it is printed below the link. Where a source said less than a draft of this page did, the sentence on this page was cut back to what the source says. Crawler documentation changes; if a quote below no longer matches its page, the page has moved on and this one will be updated.

Fetched and quoted on 16 October 2026.

  1. [1]
    AI features and your website Google Search Central · 2025-12-10 · Search engine
    To be eligible to be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet, fulfilling the Search technical requirements. There are no additional technical requirements. … Making sure that important content is available in textual form
  2. [2]
    Optimizing your website for generative AI features on Google Search Google Search Central · 2026-07-10 · Search engine
    “Chunking” content: There's no requirement to break your content into tiny pieces for AI to better understand it. Google systems are able to understand the nuance of multiple topics on a page and show the relevant piece to users.
  3. [3]
    Introduction to robots.txt Google Search Central · 2025-12-10 · Search engine
    A robots.txt file tells search engine crawlers which URLs the crawler can access on your site. This is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google.
  4. [4]
    Google's common crawlers Google Crawling Infrastructure · 2026-10-15 · Crawler operator
    Google-Extended does not impact a site's inclusion in Google Search nor is it used as a ranking signal in Google Search.
  5. [5]
    Top ways to ensure your content performs well in Google's AI experiences on Search Google Search Central Blog · 2025-05-21 · Search engine
    This includes making sure that Googlebot isn't blocked, the page works (Google receives an HTTP 200 (success) status code), and the page has indexable content.
  6. [6]
    Latest Google Search documentation updates — FAQ rich result Google Search Central · 2026-05-08 · Search engine
    Deprecating the FAQ rich result feature … This feature will no longer appear in Google Search starting May 7, 2026.
  7. [7]
    Understand the JavaScript SEO basics Google Search Central · 2026-03-04 · Search engine
    Google processes JavaScript web apps in three main phases: Crawling, Rendering, Indexing
  8. [8]
    The rise of the AI crawler Vercel, with MERJ · 2024-12-17 · Industry
    The data indicates that while ChatGPT and Claude crawlers do fetch JavaScript files (ChatGPT: 11.50%, Claude: 23.84% of requests), they don't execute them.
  9. [9]
    Overview of OpenAI Crawlers OpenAI · Crawler operator
    Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links. … Because these actions are initiated by a user, robots.txt rules may not apply.
  10. [10]
    Does Anthropic crawl data from the web, and how can site owners block the crawler? Anthropic · 2026-04-07 · Crawler operator
    Claude-SearchBot navigates the web to improve search result quality for users. … Anthropic’s Bots respect “do not crawl” signals by honoring industry standard directives in robots.txt.
  11. [11]
    Perplexity Crawlers Perplexity · Crawler operator
    PerplexityBot is designed to surface and link websites in search results on Perplexity. … Since a user requested the fetch, this fetcher generally ignores robots.txt rules.
  12. [12]
    Common Crawl FAQ — CCBot Common Crawl · Crawler operator
    CCBot is an automated crawler, checking first the robots.txt … Currently, JavaScript is not executed and Cookies are not used.
  13. [13]
    RFC 9309: Robots Exclusion Protocol IETF · 2022-09 · Standard
    This document specifies the rules originally defined by the "Robots Exclusion Protocol" [ROBOTSTXT] that crawlers are requested to honor when accessing URIs. These rules are not a form of access authorization.
  14. [14]
    Bing Webmaster Guidelines Microsoft Bing · Search engine
    Bing and Copilot search experiences rely on the same core crawling, indexing, and ranking foundation as traditional search. … URLs are more likely to be selected for grounding queries and citations when content stands on its own.
  15. [15]
    IndexNow IndexNow (Microsoft Bing, Naver, Seznam.cz, Yandex, Yep) · Standard
    In its simplest form, IndexNow is a simple ping so that search engines know that a URL and its content has been added, updated, or deleted, allowing search engines to quickly reflect this change in their search results.
  16. [16]
    Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks Lewis et al., NeurIPS 2020 (arXiv:2005.11401) · 2020-05-22 · Research
    the non-parametric memory is a dense vector index of Wikipedia, accessed with a pre-trained neural retriever. … Each Wikipedia article is split into disjoint 100-word chunks, to make a total of 21M documents.
  17. [17]
    Lost in the Middle: How Language Models Use Long Contexts Liu et al., TACL 2024 (arXiv:2307.03172) · 2023-07-06 · Research
    performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts
  18. [18]
    Introducing Contextual Retrieval Anthropic · 2024-09-19 · Industry
    In traditional RAG, documents are typically split into smaller chunks for efficient retrieval. While this approach works well for many applications, it can lead to problems when individual chunks lack sufficient context.
  19. [19]
    Chunking Strategies for LLM Applications Pinecone · 2025-06-28 · Industry
    As a rule of thumb, if the chunk of text makes sense without the surrounding context to a human, it will make sense to the language model as well.
  20. [20]
    Splitting recursively — text splitter guide LangChain · Tooling
    It tries to split on them in order until the chunks are small enough. The default list is ["\n\n", "\n", " ", ""]. … chunk_overlap: Target overlap between chunks.
  21. [21]
    Extend Claude with skills Claude Code Docs (Anthropic) · Tooling
    Skills extend what Claude can do. Create a SKILL.md file with instructions, and Claude adds it to its toolkit. Claude uses skills when relevant, or you can invoke one directly with /skill-name.
  22. [22]
    Build skills OpenAI (ChatGPT Learn — Codex) · Tooling
    Codex reads skills from repository, user, admin, and system locations. … For repositories, Codex scans `.agents/skills` in every directory from your current working directory
  23. [23]
    Agent Skills OpenCode · Tooling
    Create one folder per skill name and put a `SKILL.md` inside it. OpenCode searches these locations: … `~/.config/opencode/skills/<name>/SKILL.md` … `.claude/skills/<name>/SKILL.md` … `.agents/skills/<name>/SKILL.md`
  24. [24]
    Skills System Nous Research (Hermes Agent) · Tooling
    All skills live in `~/.hermes/skills/` — the primary directory and source of truth.
  25. [25]
    GLM Coding Plan — Overview Z.ai · Tooling
    The plan can be applied to agent tools such as ZCode, Claude Code, Codex, OpenCode, AutoClaw, and OpenClaw
  26. [26]
    Use skills in Claude Claude Help Center (Anthropic) · 2026-09-22 · Tooling
    Skills require the code execution environment to function. … Upload a ZIP file containing your skill folder.
  27. [27]
    Create and edit files with Claude Claude Help Center (Anthropic) · 2026-08-06 · Tooling
    To give Claude access to external data sources, toggle Allow network egress on … Allow network egress to package managers and specific domains: Claude can access package managers plus additional domains you specify.
  28. [28]
    Agent Skills Claude Platform Docs (Anthropic) · Tooling
    Depending on user/admin settings, Skills may have full, partial, or no network access. … Skills that fetch data from external URLs pose particular risk, as fetched content may contain malicious instructions.

Other free AI search tools