What an AI assistant reads when it reads your page

A retrieval system never reads your page. It reads passages of it — a handful, chosen by similarity to a question — and everything else on the page might as well not exist for that answer. This is not a detail of one product's implementation. It is what retrieval-augmented generation is, and has been since the paper that named it in 2020: retrieve passages, then write from them.

So the interesting question about a page is not how good it is. It is which passages a retriever would receive, in what order, and whether the one holding your answer is among them.

We built Vectorscope to answer that for one URL at a time, and it runs the real pipeline — the same chunker, the same embedding model, the same passage cap the paid product applies. A demo running a simplified version would flatter the pipeline it exists to demonstrate.

Six pages, and the number that turned out not to matter

We ran six pages: a vendor's guide to a tax form, three reference and documentation pages, a very long encyclopedia article, and our own Vectorscope page. Every figure below came out of the tool on 19 August 2026, against the deployed pipeline — voyage-4-lite embeddings at 1,024 dimensions, and a ceiling of 26 passages a page.

The number we expected to be the story was the fourth column: how much of a page never reaches the index. On five of six pages it was zero. Not close to zero — zero. Nothing was lost, because none of those pages was long enough to exceed the ceiling.

It bites exactly once, and it bites hard. The Wikipedia article on transformers is 14,163 words; the retriever received 26 passages holding 47,275 characters and left 53,903 characters unread — 53.3% of the page. Over half of the most-linked explanation of the architecture behind every model mentioned here is, as far as one retrieval pass is concerned, not there.

The lesson is narrower than the usual advice and more useful: page length is not a retrieval problem until it is enormous, and then it is a severe one. Below roughly five thousand words, the ceiling is not what is costing you.

What did differ: how much a page repeats itself

Look at the last column instead. Self-similarity is the average cosine similarity between a page's own passages — how alike the parts of one page are to each other. The six pages span 0.698 to 0.854, and the order is not the order of their word counts.

A low number means the page covers ground. The Wikipedia SEO article at 0.708 has six groups, one of them a single passage on international markets that nothing else on the page resembles — that is a page with a genuinely separate section, and a retriever asked about international SEO has something specific to return.

A high number means the page says one thing many times. Which brings us to the uncomfortable part.

We ran it on ourselves and came last

Our own Vectorscope page scored 0.854 — the highest, by a distance, on six pages we did not choose to flatter ourselves. Its 1,556 words became six passages in three groups, one of which is a single passage and another of which holds four. In plain terms: the page makes one argument, repeatedly, in slightly different words.

Worse, 1,994 of its 10,235 extracted characters — 19% — were classified as utility rather than main content: navigation, controls, the strip of chips at the top. On the W-9 guide the equivalent figure was 434 characters out of 29,270, which is 1.5%.

We are not going to pretend this is a subtle finding. The page has since had eight sections of real explanation added underneath the tool, and the reading above was taken after that; before them it would have been worse. It is the clearest demonstration we have that the measure is worth having: it told us something true that we did not want to hear, about the one page we look at most.

The three ways to cut a page, and why the middle one matters most

Chunking is the step nobody sees and everybody is judged on.

Fixed-length chunking cuts every N characters. It is the default in most pipelines because it is trivial to implement, and it cuts through the middle of sentences, separates a heading from the answer beneath it, and produces passages that begin with half a word. It is fast, cheap, repeatable, and the reason many pages retrieve badly for questions they answer perfectly.

Heading-aware chunking uses the page's own H2 and H3 structure as the cut lines, so a passage arrives carrying the heading that names it. This is the single largest improvement available to most pages, and it has a consequence people miss: your heading text is now doing retrieval work. A section called "Overview" contributes nothing to the passage beneath it. A section called "How much does it cost?" contributes the question itself.

Semantic chunking compares neighbouring blocks and cuts where the meaning shifts, whether or not a heading is there. It produces passages that stand alone, and it costs an embedding pass over the page before any retrieval happens.

None of the three is free of judgement. A cut is a decision about where one idea ends, and Vectorscope shows you the decisions rather than a score, because a score would hide them.

What it cannot tell you

Three things, and we would rather say them here than have somebody discover them and conclude the tool was overselling itself.

It cannot tell you whether any specific assistant has your page in its index. Nobody outside those companies can see that, and a tool claiming otherwise is inferring from something it will not name.

It cannot tell you how a model ranks your page against a competitor's. That needs a corpus of both pages, which is the product rather than the demo.

It cannot tell you whether you were cited in an answer today. That is a measurement over time, on prompts you choose, and it is a different instrument.

What it can do is show you, exactly, the passages a retriever receives from one URL — which is the part of the chain everybody guesses at.

If you build retrieval rather than content

The same page works as a reference implementation you can point at any URL:

• Run a page you already index, and compare the passage boundaries with your chunker's.
• Compare extraction ratios on the same URL. If yours is lower, your extractor is discarding body text, not boilerplate.
• Ask the page a question whose answer you know, and check that the top passage is the one containing it.
• Run a page that is far too long. The passages beyond the cap are the ones your own top-k will never reach either.

Sources, and where to read further

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — the paper the pattern is named after.
• Cosine similarity — the measure behind every claim that two passages are about the same thing.
• Google Search Central on AI features and your site and the SEO Starter Guide, one of the six pages read above.
• Google's introduction to structured data and schema.org, for the machine-readable half.
• llms.txt — the convention we follow at /llms.txt, which describes this tool in the one format a model does not have to parse.
• OpenAI's embeddings guide, if the word embedding is new.

Our own methodology page carries every formula the product uses and what each number is allowed to claim.