Duplicates and canonicals
Describes GLLmolfar 0.9.35current release · checked 8 Nov 2026
Three ways a page can repeat another — the same text, almost the same words, or the same meaning in other words — and the canonicals that say which address should count. Each is a filter; the closest page and the similarity sit beside each row.
- Where
- Site › All pages › More columns… › Content and Canonicals
- Mode
- Pro
- Built from
- The text of each page; near duplicates by meaning also need the embeddings
- Changes
- Exact and text duplicates live; by meaning when the crawl's analysis ends
On the screen
The Content column set on a duplicates filter, with a page picked and its Similar pages open in the inspector.
Canonicals. The column set for canonical tags. The duplicate filters live in the Content set; on a site that has duplicates they list the pages with their count.
Filter. Here Self-referencing (500) · 100%: each page's canonical is its own address. Other filters list pages canonicalised elsewhere, canonicals to a bad target, and pages without one.
Canonical, Indexability, Status. The address each page names as canonical, whether the page can be indexed, and its status.
URL details. The picked page's canonical and indexability again, beside the rest of its details.
Three kinds of duplicate
| Filter | What matches | Needs |
|---|---|---|
| Exact duplicates | The page's main text, menus and footers removed, is identical to another page's (compared by a hash). | Nothing more |
| Near duplicates by text | Almost the same words: 90% or more of their 5-word sequences are shared (Jaccard, estimated from a signature). | A crawl made by 0.9.35 or later |
| Near duplicates by meaning | The pages say almost the same thing, in any words: cosine 0.94 or more between their embeddings. The same pairs as Cannibalization. | Embeddings |
Near duplicates by text leave out pairs that are already exact duplicates, two addresses that differ only by tracking parameters, pages of one paginated list, and translations of one page. Canonicalised pages are not compared. Pages from a crawl saved before 0.9.35 carry no signature and are left out, not guessed.
| Column | What it shows |
|---|---|
| Words, Text ratio, % | Words of main text, and text as a share of the HTML. |
| Topic | The page's topic, from the embeddings. |
| Closest page by text | The page with the most similar words, among those at 90% or more. |
| Text similarity | How similar, in percent. |
| JS, Size, KB, Indexability | Whether the page was rendered, its HTML size, and whether it can be indexed. |
Canonicals
| Filter | Which pages |
|---|---|
| Has a canonical | The page names a canonical URL. |
| Self-referencing | The canonical is the page's own address. |
| Canonicalised to another URL | The canonical points elsewhere, so the page is not indexable as itself. |
| Missing | No canonical at all. |
| Points to a non-indexable URL | The canonical names a crawled page that is not 200 or not indexable. |
Reports › Canonical problems puts these into one file. A canonical changed by JavaScript is a filter of the JavaScript column set.
Addresses with parameters
Copies made by sort, filter or session parameters are best kept out of the crawl. Settings › Crawl scope › URLs with parameters: Crawl as found, Remove the parameters, or Skip them; with Crawl as found, Parameters to remove names the ones to strip and keeps the others. Tracking parameters such as utm_* are handled on their own either way.