# Crawl a list of URLs

> Three ways to say which pages a crawl reads: a list of addresses and nothing else, extra addresses added to a crawl that is running, or a crawl kept inside one folder of the site.

Describes GLLmolfar 0.9.35, checked 2026-11-08. Canonical page: https://www.getlooploop.com/gllmolfar/docs/list-crawl

- **Where:** Crawl › Crawl a list of URLs… (Ctrl+L); Crawl › Add URLs to this crawl…; Settings › Crawl scope
- **Mode:** Simple and Pro
- **Built from:** The addresses you give
- **Changes:** When the crawl starts

## On the screen

The dialog takes addresses one per line, or a file. Here, pages of getlooploop.com pasted in. (The dialog's first line, which names the account level's URL limit, is left out of this picture.)

![The Crawl a list of URLs dialog of GLLmolfar with seven addresses of getlooploop.com pasted in](https://www.getlooploop.com/images/gllmolfar/docs/list-crawl.webp)

Numbered parts of the screenshot:

1. **The list.** One address per line; here seven pages of getlooploop.com. Anything that looks like an address in pasted text is taken; duplicates count once.
2. **Or a file.** A text, CSV or XML file; its addresses are added to what is pasted.
3. **Start crawl.** Crawls these addresses and only these.

## A list of URLs

- Only these URLs are crawled: the crawl stops when the list is done. Their links are recorded on each page as in any crawl.
- Addresses must start with `http://` or `https://`. With none, the app says: **No URLs found. Paste full addresses starting with http:// or https://.**
- A list longer than the account level's limit is cut, and a message says how many were taken of how many.
- Typical lists: the pages Search Console names under [Lost clicks](https://www.getlooploop.com/gllmolfar/docs/gsc-lost-clicks), a sitemap's addresses, the pages a developer changed.
- The toolbar's mode selector has **List of URLs** too; Start then asks for the list.

## Add URLs to this crawl

While a crawl runs, **Crawl › Add URLs to this crawl…** takes more addresses, one per line. **They are crawled next, even outside the include and exclude rules.** The toast says how many were added to the queue.

## One folder of a site

To crawl only `/blog/`, start at `https://site.com/blog/` and turn on **Stay inside the start folder** in [Settings › Crawl scope](https://www.getlooploop.com/gllmolfar/docs/settings-scope). Links, redirects and sitemap addresses outside the folder are left out and counted as excluded; the start URL and URLs you add are always crawled. An **Include** line (`/blog/`) does the same with a regular expression, for any pattern.

> **Which folder.** The path up to its last `/`: `/blog/post.html` stays in `/blog/`. A last part without a dot reads as a folder, so a start at `/blog` stays in `/blog/`. The setting has no effect in list mode, where only the list is crawled anyway.

## States and messages

- **Empty list** — the app says: "No URLs found. Paste full addresses starting with http:// or https://." Nothing in the box or the file looked like an address.
- **Not running** — the app says: "Add URLs to this crawl…" Available only while a crawl runs.

## Related pages

- [The crawl screen](https://www.getlooploop.com/gllmolfar/docs/crawl-screen.md): What GLLmolfar shows while a crawl runs: four stages, the crawl and fragment cards, robots.txt rules, the run charts, the last pages read and CAPTCHA pauses.
- [Settings › Crawl scope](https://www.getlooploop.com/gllmolfar/docs/settings-scope.md): Settings › Crawl scope: which URLs a crawl may visit — include and exclude patterns, depth, parameters, the start folder, languages and links.
- [Search Console: Lost clicks](https://www.getlooploop.com/gllmolfar/docs/gsc-lost-clicks.md): GSC › Lost clicks: pages that lost Google clicks over the last 28 days, the reason the numbers point at, the lost queries, and what the crawl found there.
- [History](https://www.getlooploop.com/gllmolfar/docs/history.md): The History tab: every saved crawl of a site, a trend of one number across them, and a comparison of two crawls page by page and check by check.