# The crawl screen

> While a crawl runs, the Overview tab shows the run itself: how far it is, what each stage is doing and the last pages read. It stays until the fragments and the topics are done, then the Overview takes its place.

Describes GLLmolfar 0.9.22, checked 2026-11-01. Canonical page: https://www.getlooploop.com/gllmolfar/docs/crawl-screen

- **Where:** Overview, during a run
- **Mode:** Simple and Pro
- **Built from:** The crawl in progress
- **Changes:** Every second

## On the screen

![The crawl screen of GLLmolfar during a crawl of getlooploop.com](https://www.getlooploop.com/images/gllmolfar/docs/crawl-screen.webp)

Numbered parts of the screenshot:

1. **Stages.** Crawl, Fragments, Topics and cannibalisation, Done. The stage in progress is marked; a finished one gets a tick. Done shows the forecast end time.
2. **Views.** **Overview**, **Neuro-pipeline** and **Terminal** show the same run three ways. **Compact** keeps each card to its numbers and the load chart as a small strip.
3. **Crawl card.** Pages read out of the target, the rate in pages a second, addresses found against the limit, and the time since the start.
4. **Fragments card.** Fragments embedded out of those found, fragments a second, the queue, and when the fragments end after the crawl. Two bars: first fragments, which give the topics, and more fragments, which refine them.
5. **Rules of the run.** Who the site sees (the user agent, to copy for a log search, and this computer's address), what robots.txt allows, and in Balanced mode the next checkpoint.
6. **Page checks.** Critical, warnings and info found so far, with the most frequent checks under each. The counts grow as the crawl reads.
7. **The whole run.** One chart of pages read and fragments waiting, with the forecast at the current pace. Below it: every fragment as a cell, the last pages read, and the load by stage.

## The four stages

| Stage | While it runs | When it is done |
| --- | --- | --- |
| Crawl | N / target · percent | N pages · time taken |
| Fragments | alongside · percent of those found, then embedding · percent | N fragments |
| Topics and cannibalisation | after the fragments, then building the topics… | N topics · N pairs |
| Done | ≈ the forecast end time | a tick |

Fragments are the pieces of page text the model on this computer turns into vectors. Topics and cannibalisation pairs are built from them, so the third stage waits for the second.

## Cards and charts

| Block | What it shows |
| --- | --- |
| Crawling the site | Pages read **of N pages**, the pages left, the rate, **Addresses found** with the limit, and the time since the start. |
| Fragments | Fragments embedded **of N fragments**, fragments a second, and the queue: growing when pages come faster than the model reads them, falling, or empty. |
| robots.txt | Whose rules the crawl follows (the site's, your own from Settings, or none), the pace, the number of streams, closed paths, sitemaps, and the AI bots the site shuts out. |
| Balanced | In Balanced mode: the next checkpoint, or the one under way, where the crawl waits for the embeddings and saves the topics so far. |
| Page checks | Findings at three levels (critical, warnings, info) filling as the crawl reads, the most frequent checks under each. When the crawl ends, a check opens its pages. |
| The whole run | One chart: pages read, fragments waiting, the forecast at the current pace, and the moment the crawl completed. |
| Fragments mosaic | Each fragment as a cell: first fragments, more fragments, in the queue, and pages not found yet. |
| Last pages read | The last eight addresses with their status code, response time and a note: rendered, redirect or CAPTCHA. Click or Enter opens the page in the browser. |
| Load by stage | Pages a second for each stage over the run. The lines start after the first seconds. |

## Views and Compact

- **Overview** — the cards and charts above.
- **Neuro-pipeline** — the same run drawn as a network of its stages.
- **Terminal** — the same run drawn as a terminal, with the topics layer as tiles of dots.
- **Compact** — in Overview only: each card keeps its numbers, the fragments shrink to a strip, and the run chart and **Load by stage** stay as small, low strips: the stages' pace and what held them, no more. The run fits beside other work, and the app remembers the choice.

## When the site asks for a CAPTCHA

A site may answer with a bot check instead of its pages — in place, or by sending every page to a check page. The crawl then pauses, and a window opens with the site's check for you to pass. A bar over the app says **The crawl is paused: the site asks for a CAPTCHA.** Once the site lets you through, the crawl goes on by itself, with the check passed.

| Button on the bar | What it does |
| --- | --- |
| Show the window | Brings the check's window back to the front. |
| I passed it, continue | Tries the page again and resumes the crawl now. |
| Skip checks in this crawl | Goes on without stopping at checks again; pages that answer with one are noted as CAPTCHA. Closing the check's window does the same. |

[Settings › Crawling](https://www.getlooploop.com/gllmolfar/docs/settings-crawling) holds the switch, on by default: **At a CAPTCHA, pause and open it for me to pass**. Off, the crawl does not stop, and such pages are counted as met with a CAPTCHA.

## After the crawl

When the crawl ends but the model is still reading fragments, the screen says **Stage 2 of 2 · finishing**: pages crawled and the time, fragments read out of the total, and about how long is left. **Ready now** lists the fixes that do not wait for the fragments.

- **Embeddings off** — the app says: "Embeddings are off" Without them there are no topics, cannibalisation pairs or meaning search. Switch them on in the toolbar; the queue keeps what is waiting.
- **No score yet** — the app says: "Scored when the crawl ends: the checks need every page." The score waits for the last page, so the screen does not guess it.
- **Limit** — the app says: "This crawl stops at N URLs; M found so far." The crawl reached the URL limit set when it started.

## Related pages

- [Overview](https://www.getlooploop.com/gllmolfar/docs/overview.md): The Overview tab after a crawl: the health score, six counters, what to fix first, response codes, indexability, site depth, topics and earlier crawls.
- [Settings › Storage](https://www.getlooploop.com/gllmolfar/docs/storage.md): Settings › Storage: how much disk the saved crawls, the embedding model and downloaded updates take, per project, and the actions that free it.
- [Reports](https://www.getlooploop.com/gllmolfar/docs/reports.md): The Reports tab: eleven reports in four directions, each answering one question about the open crawl, with a language filter and HTML or PDF export.
- [Settings › Crawling](https://www.getlooploop.com/gllmolfar/docs/settings-crawling.md): Settings › Crawling: request speed, connections, the user agent, the timeout, robots.txt, a pause at a CAPTCHA, JavaScript rendering and service pages.