Documentation menu

Settings › Crawl scope

Describes GLLmolfar 0.9.22current release 0.9.23 · checked 1 Nov 2026

Which addresses a crawl may visit. Patterns keep or drop URLs, a depth stops it some clicks from the start, and three switches decide about subdomains, nofollow links and external links.

Where
Settings › Crawl scope
Opens
The gear at the top right
Stored
In the settings file on this computer
Applies
To the next crawl

On the screen

Settings › Crawl scope in GLLmolfar
real screenshotProject getlooploop.com. Other sites and brands on screen are replaced with neutral names.

  1. Include. Only URLs that match one of these lines are crawled.

  2. Exclude. URLs that match any of these lines are skipped.

  3. Max crawl depth. Clicks from the start URL; 0 is no limit.

  4. URLs with parameters. Crawl as found, Remove the parameters or Skip them.

  5. Subdomains. Crawl subdomains as part of the site.

  6. Nofollow. Follow internal nofollow links.

  7. External links. Check external links.

Fields

FieldWhat it doesOptionsDefault
Include (regex, one per line)When any line is given, a URL must match one of them to be crawled. Example: /blog/.Regular expressionsEmpty
Exclude (regex, one per line)A URL that matches any line is skipped. Example: \?sort=|/tag/.Regular expressionsEmpty
Max crawl depth (0 = no limit)How many clicks from the start URL the crawl follows.0 to 1000
URLs with parametersWhat to do with an address that has a query string (?…).Crawl as found; Remove the parameters; Skip themCrawl as found
Crawl subdomains as part of the siteTreats subdomains of the start host, such as blog.example.com, as the same site.On or offOff
Follow internal nofollow linksFollows internal links marked rel="nofollow".On or offOn
Check external linksRequests each external link once and records its status.On or offOn

States and messages

Bad patternThis line is not a valid regular expression: {detail}

Save is refused; the message shows under the switches and the Include box gets the focus.

Next crawlCrawl and embedding settings apply to the next crawl.

A crawl that is running keeps the scope it started with.

Related pages

Settings › CrawlingSettings › Crawling: request speed, connections, the user agent, the timeout, robots.txt, a pause at a CAPTCHA, JavaScript rendering and service pages.Settings › RequestsSettings › Requests: extra HTTP headers sent with every page request, and a user name and password for a staging site behind Basic authentication.Settings › GeneralSettings › General: the interface language, term explanations, the interface mode, the default crawl size and mode, updates, statistics, the tray, MCP, sound.The crawl screenWhat GLLmolfar shows while a crawl runs: four stages, the crawl and fragment cards, robots.txt rules, the run charts, the last pages read and CAPTCHA pauses.