# Settings › Crawling

> How the crawler asks a site for its pages: how fast, as which user agent, how long it waits, which robots.txt it obeys, what it does at a CAPTCHA, and when it renders JavaScript.

Describes GLLmolfar 0.9.22, checked 2026-11-01. Canonical page: https://www.getlooploop.com/gllmolfar/docs/settings-crawling

- **Where:** Settings › Crawling
- **Opens:** The gear at the top right
- **Stored:** In the settings file on this computer
- **Applies:** To the next crawl

## On the screen

![Settings › Crawling in GLLmolfar](https://www.getlooploop.com/images/gllmolfar/docs/settings-crawling.webp)

Numbered parts of the screenshot:

1. **Speed.** **Requests per second** and **Parallel connections**.
2. **User-Agent.** Who the crawler says it is: GLLmolfar, a search engine bot, Chrome, an AI crawler or your own.
3. **Respect robots.txt.** Obey the site's robots.txt.
4. **At a CAPTCHA.** **At a CAPTCHA, pause and open it for me to pass**: the crawl waits while you pass the site's check, then goes on.
5. **Custom robots.txt.** A robots.txt to use instead of the site's, and **Test URL** to try an address against it.
6. **JavaScript rendering.** Off, pages with little text, or all pages.
7. **Service pages.** **Count service pages in content checks**.

## Fields

| Field | What it does | Options | Default |
| --- | --- | --- | --- |
| Requests per second | The most requests the crawler sends each second. | 0.2 to 20 | 5 |
| Parallel connections | How many requests run at once. | 1 to 16 | 4 |
| User-Agent | The name the crawler gives the site. A bot preset obeys that bot's own group in robots.txt. | GLLmolfar; Googlebot Smartphone; Bingbot; Chrome (Windows); under AI: GPTBot (OpenAI), OAI-SearchBot (OpenAI), ChatGPT-User (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity); Custom… | GLLmolfar |
| Custom… (text box) | Shown for **Custom…**: your own user agent. It cannot name another company: Save is refused, the box gets the focus, and the section says **A custom user agent cannot name another company**, with the word it found. | Text | Empty |
| Request timeout, s | Seconds to wait for one answer before giving up. | 3 to 120 | 15 |
| Respect robots.txt | Skips addresses the site's robots.txt closes. With a bot preset as the user agent, robots.txt is obeyed even when this is off. | On or off | On |
| At a CAPTCHA, pause and open it for me to pass | When a page answers with a CAPTCHA or a bot check, in place or by sending every page to a check page, the crawl pauses and the page opens in a window of its own, with the crawl's user agent. Once the site lets you through, the crawl resumes and asks for the page again with the cookies the check left. Off, and in planned crawls: such pages are recorded as bot checks and the crawl goes on. | On or off | On |
| Custom robots.txt | **Used instead of the site's robots.txt, to try changes before they go live. Leave empty to use the site's.** | Text | Empty |
| Test URL | Checks one address against the custom robots.txt, or against the site's when the box is empty. Without an address it tests the one in the toolbar. | Button | — |
| JavaScript rendering | Whether pages are opened in a browser so that text written by JavaScript is seen. | Off · raw HTML only; Auto · render pages with little text; All pages (slower) | Auto · render pages with little text |
| Count service pages in content checks | **Off: privacy, terms, login, cart, the HTML sitemap and similar pages are left out of the author, date, thin content, cannibalization and other content checks. Each check says how many it left out.** | On or off | Off |

## States and messages

- **Allowed** — the app says: "Allowed · custom robots.txt" **Test URL**: the address may be crawled. The second part says which robots.txt answered: **custom robots.txt**, **the site's robots.txt** or **the site has no robots.txt**.
- **Blocked** — the app says: "Blocked · the site's robots.txt" **Test URL**: the robots.txt closes this address.
- **Next crawl** — the app says: "Crawl and embedding settings apply to the next crawl." A crawl that is running keeps the settings it started with.

## When a site asks for a CAPTCHA

With the switch on, the first check a crawl meets pauses it and opens a window titled **Pass the site's check — GLLmolfar**. A bar in the main window says what happened and offers three buttons.

- **Paused** — the app says: "The crawl is paused: the site asks for a CAPTCHA." Under it: **Pass it in the window that opened; the crawl goes on by itself once the site lets you through.** When the app recognises the check, its maker's name comes first.

| Button | What it does |
| --- | --- |
| Show the window | Brings the check's window to the front. |
| I passed it, continue | Resumes the crawl when the window has not noticed on its own. |
| Skip checks in this crawl | Closes the window; this and later checks are recorded as bot checks until the crawl ends. Closing the check's window does the same. |

## Related pages

- [Settings › Crawl scope](https://www.getlooploop.com/gllmolfar/docs/settings-scope.md): Settings › Crawl scope: which URLs a crawl may visit — include and exclude patterns, depth, URLs with parameters, subdomains, nofollow and external links.
- [Settings › Requests](https://www.getlooploop.com/gllmolfar/docs/settings-requests.md): Settings › Requests: extra HTTP headers sent with every page request, and a user name and password for a staging site behind Basic authentication.
- [Settings › General](https://www.getlooploop.com/gllmolfar/docs/settings-general.md): Settings › General: the interface language, term explanations, the interface mode, the default crawl size and mode, updates, statistics, the tray, MCP, sound.
- [The crawl screen](https://www.getlooploop.com/gllmolfar/docs/crawl-screen.md): What GLLmolfar shows while a crawl runs: four stages, the crawl and fragment cards, robots.txt rules, the run charts, the last pages read and CAPTCHA pauses.