# Settings › Crawl scope

> Which addresses a crawl may visit. Patterns keep or drop URLs, a depth stops it some clicks from the start, and three switches decide about subdomains, nofollow links and external links.

Describes GLLmolfar 0.9.22, checked 2026-11-01. Canonical page: https://www.getlooploop.com/gllmolfar/docs/settings-scope

- **Where:** Settings › Crawl scope
- **Opens:** The gear at the top right
- **Stored:** In the settings file on this computer
- **Applies:** To the next crawl

## On the screen

![Settings › Crawl scope in GLLmolfar](https://www.getlooploop.com/images/gllmolfar/docs/settings-scope.webp)

Numbered parts of the screenshot:

1. **Include.** Only URLs that match one of these lines are crawled.
2. **Exclude.** URLs that match any of these lines are skipped.
3. **Max crawl depth.** Clicks from the start URL; 0 is no limit.
4. **URLs with parameters.** **Crawl as found**, **Remove the parameters** or **Skip them**.
5. **Subdomains.** **Crawl subdomains as part of the site**.
6. **Nofollow.** **Follow internal nofollow links**.
7. **External links.** **Check external links**.

## Fields

| Field | What it does | Options | Default |
| --- | --- | --- | --- |
| Include (regex, one per line) | When any line is given, a URL must match one of them to be crawled. Example: `/blog/`. | Regular expressions | Empty |
| Exclude (regex, one per line) | A URL that matches any line is skipped. Example: `\?sort=\|/tag/`. | Regular expressions | Empty |
| Max crawl depth (0 = no limit) | How many clicks from the start URL the crawl follows. | 0 to 100 | 0 |
| URLs with parameters | What to do with an address that has a query string (`?…`). | Crawl as found; Remove the parameters; Skip them | Crawl as found |
| Crawl subdomains as part of the site | Treats subdomains of the start host, such as blog.example.com, as the same site. | On or off | Off |
| Follow internal nofollow links | Follows internal links marked `rel="nofollow"`. | On or off | On |
| Check external links | Requests each external link once and records its status. | On or off | On |

> **What the rules leave alone.** The start URL and URLs you add yourself are never dropped by include, exclude, parameters or depth. Parameters are handled first, so with **Remove the parameters** the patterns see the address without its query string.

## States and messages

- **Bad pattern** — the app says: "This line is not a valid regular expression: {detail}" Save is refused; the message shows under the switches and the Include box gets the focus.
- **Next crawl** — the app says: "Crawl and embedding settings apply to the next crawl." A crawl that is running keeps the scope it started with.

## Related pages

- [Settings › Crawling](https://www.getlooploop.com/gllmolfar/docs/settings-crawling.md): Settings › Crawling: request speed, connections, the user agent, the timeout, robots.txt, a pause at a CAPTCHA, JavaScript rendering and service pages.
- [Settings › Requests](https://www.getlooploop.com/gllmolfar/docs/settings-requests.md): Settings › Requests: extra HTTP headers sent with every page request, and a user name and password for a staging site behind Basic authentication.
- [Settings › General](https://www.getlooploop.com/gllmolfar/docs/settings-general.md): Settings › General: the interface language, term explanations, the interface mode, the default crawl size and mode, updates, statistics, the tray, MCP, sound.
- [The crawl screen](https://www.getlooploop.com/gllmolfar/docs/crawl-screen.md): What GLLmolfar shows while a crawl runs: four stages, the crawl and fragment cards, robots.txt rules, the run charts, the last pages read and CAPTCHA pauses.