A collection of text documents that an AI-search system indexes or trains on to generate answers.
Marketers concerned with optimizing content for AI-search results and web indexing.
01What it is and how it works
The corpus is built by crawling the web (or a proprietary feed) and storing the retrieved pages in a searchable index. When a user asks a question, the AI‑search engine first retrieves the most relevant passages from this corpus using vector or keyword matching, then feeds those passages to a language model that synthesizes the final answer. Updates to the corpus happen continuously as new pages are discovered, old pages are re‑crawled, or low‑quality content is dropped.
A corpus is just the set of pages or documents the AI looks at when it answers a question.
02What to do about it
Marketers can shape the corpus that powers AI‑search results by ensuring their site’s content is crawlable, valuable, and up‑to‑date. This week, run a crawl audit with Google Search Console’s URL Inspection tool to spot blocked pages, publish a fresh XML sitemap that includes new product pages, add schema.org markup to help the engine understand the content, and remove or noindex thin or duplicate pages that could dilute the corpus quality.
03How it is measured or noticed
You can gauge the health of your corpus by looking at index coverage reports (which show how many submitted URLs are indexed), crawl stats (which reveal how often Googlebot visits your site), and the Search Performance report (which indicates which queries are pulling impressions from your pages). In AI‑search specific tools, check the retrieval recall metric if available, or monitor changes in answer‑source attribution to see whether your content is being selected as a source.
04Common mistakes
- Allowing duplicate or boilerplate content to stay in the corpus, which can confuse the retrieval model.
- Blocking essential resources like CSS or JavaScript via robots.txt, causing the crawler to see a broken version of the page.
- Neglecting to update the corpus after a site migration, leading to outdated or missing information in AI answers.
- Relying solely on paid search ads to drive visibility while ignoring organic corpus quality.
05Limits
The concept of a corpus mainly applies to retrieval‑augmented AI search; it is less relevant for pure generative models that generate answers without consulting external documents. It is also often confused with the term ‘dataset’ (which refers to the training data for the language model itself) or with the ‘index’ (the internal data structure used for fast lookup), but the corpus is the raw set of documents before indexing.
06Worked example
Imagine a retailer adds a new line of eco‑friendly sneakers. After publishing the product pages, they submit an updated sitemap and verify that Googlebot crawls the URLs within two days. The newly crawled pages enter the corpus, and when a user asks ‘What are the best sustainable sneakers for running?’ the retrieval step can now pull the fresh product descriptions, allowing the AI‑search answer to include the new line.
"The new sneaker pages appeared in the answer within 48 hours of sitemap submission."
Frequently asked questions
What is a corpus in AI search?
A corpus is a collection of text documents that an AI-search system indexes or trains on to generate answers. It determines what the model can retrieve and influences the relevance of its responses.
How is a corpus built?
A corpus is built by crawling the web or a proprietary feed and storing the retrieved pages in a searchable index.
Can marketers influence the corpus?
Yes, marketers can shape the corpus by ensuring their site’s content is crawlable, valuable, and up-to-date.
How do you measure the health of a corpus?
You can gauge the health of your corpus by looking at index coverage reports and crawl stats.
Does a corpus apply to all AI models?
No, the concept of a corpus mainly applies to retrieval-augmented AI search and is less relevant for pure generative models.
What happens if a corpus is outdated?
An outdated corpus can lead to irrelevant search results and reduced accuracy in AI-generated answers.
Can a corpus include non-web sources?
Yes, a corpus can be built from proprietary feeds or other structured data sources in addition to web content.
Asked out loud
spoken, not typedThe same term in the words somebody uses speaking to an assistant rather than typing into a box — written from the situation, which is why each one carries the situation it came from.
A corpus is the collection of text an AI search system uses to generate answers. It determines what the model can retrieve and affects response relevance.
Yes, if your content is crawlable and valuable, it gets indexed and becomes part of the corpus that powers AI search results.
If your product pages aren’t indexed, they won’t appear in AI search results, which can hurt visibility and sales.