# Duplicates and canonicals

> Three ways a page can repeat another — the same text, almost the same words, or the same meaning in other words — and the canonicals that say which address should count. Each is a filter; the closest page and the similarity sit beside each row.

Describes GLLmolfar 0.9.35, checked 2026-11-08. Canonical page: https://www.getlooploop.com/gllmolfar/docs/duplicates-canonicals

- **Where:** Site › All pages › More columns… › Content and Canonicals
- **Mode:** Pro
- **Built from:** The text of each page; near duplicates by meaning also need the embeddings
- **Changes:** Exact and text duplicates live; by meaning when the crawl's analysis ends

## On the screen

The Content column set on a duplicates filter, with a page picked and its **Similar pages** open in the inspector.

![The Canonicals column set of GLLmolfar on a 500-page crawl of getlooploop.com, filtered to self-referencing canonicals, with the home page's URL details below](https://www.getlooploop.com/images/gllmolfar/docs/duplicates-canonicals.webp)

Numbered parts of the screenshot:

1. **Canonicals.** The column set for canonical tags. The duplicate filters live in the **Content** set; on a site that has duplicates they list the pages with their count.
2. **Filter.** Here **Self-referencing (500) · 100%**: each page's canonical is its own address. Other filters list pages canonicalised elsewhere, canonicals to a bad target, and pages without one.
3. **Canonical, Indexability, Status.** The address each page names as canonical, whether the page can be indexed, and its status.
4. **URL details.** The picked page's canonical and indexability again, beside the rest of its details.

## Three kinds of duplicate

| Filter | What matches | Needs |
| --- | --- | --- |
| Exact duplicates | The page's main text, menus and footers removed, is identical to another page's (compared by a hash). | Nothing more |
| Near duplicates by text | Almost the same words: 90% or more of their 5-word sequences are shared (Jaccard, estimated from a signature). | A crawl made by 0.9.35 or later |
| Near duplicates by meaning | The pages say almost the same thing, in any words: cosine 0.94 or more between their embeddings. The same pairs as [Cannibalization](https://www.getlooploop.com/gllmolfar/docs/cannibalization). | Embeddings |

Near duplicates by text leave out pairs that are already exact duplicates, two addresses that differ only by tracking parameters, pages of one paginated list, and translations of one page. Canonicalised pages are not compared. Pages from a crawl saved before 0.9.35 carry no signature and are left out, not guessed.

| Column | What it shows |
| --- | --- |
| Words, Text ratio, % | Words of main text, and text as a share of the HTML. |
| Topic | The page's topic, from the embeddings. |
| Closest page by text | The page with the most similar words, among those at 90% or more. |
| Text similarity | How similar, in percent. |
| JS, Size, KB, Indexability | Whether the page was rendered, its HTML size, and whether it can be indexed. |

## Canonicals

| Filter | Which pages |
| --- | --- |
| Has a canonical | The page names a canonical URL. |
| Self-referencing | The canonical is the page's own address. |
| Canonicalised to another URL | The canonical points elsewhere, so the page is not indexable as itself. |
| Missing | No canonical at all. |
| Points to a non-indexable URL | The canonical names a crawled page that is not 200 or not indexable. |

**Reports › Canonical problems** puts these into one file. A canonical changed by JavaScript is a filter of the [JavaScript](https://www.getlooploop.com/gllmolfar/docs/javascript) column set.

## Addresses with parameters

Copies made by sort, filter or session parameters are best kept out of the crawl. [Settings › Crawl scope](https://www.getlooploop.com/gllmolfar/docs/settings-scope) › **URLs with parameters**: **Crawl as found**, **Remove the parameters**, or **Skip them**; with Crawl as found, **Parameters to remove** names the ones to strip and keeps the others. Tracking parameters such as utm_* are handled on their own either way.

## Related pages

- [Cannibalization](https://www.getlooploop.com/gllmolfar/docs/cannibalization.md): The Cannibalization tab: groups of pages that say almost the same thing, the page to keep, the pages side by side and their text passage by passage.
- [Pagination](https://www.getlooploop.com/gllmolfar/docs/pagination.md): The Pagination column set: pages of paginated lists, their page numbers and rel=prev/next, and the faults of page 2 and later.
- [Settings › Crawl scope](https://www.getlooploop.com/gllmolfar/docs/settings-scope.md): Settings › Crawl scope: which URLs a crawl may visit — include and exclude patterns, depth, parameters, the start folder, languages and links.
- [JavaScript](https://www.getlooploop.com/gllmolfar/docs/javascript.md): The JavaScript column set: what differs between served HTML and the rendered page (text, title, canonical, links) and images seen only after scrolling.
- [All pages (Internal)](https://www.getlooploop.com/gllmolfar/docs/all-pages.md): The All pages tab, called Internal in Pro mode: every crawled URL in one table, with filters, search, column sets, an overview panel and a page inspector.