A directive in a robots.txt file that asks compliant crawlers to pause a specified number of seconds between successive requests.
Website administrators reading about site optimization and managing bot traffic.
01What it is and how it works
When a crawler reads a site’s robots.txt, it may encounter a line such as Crawl-delay: 10. The crawler then spaces its HTTP requests at least ten seconds apart. The directive is not part of the original robots.txt standard but was added by major search engines as a courtesy to reduce server load. Only bots that explicitly support the directive honor it; others ignore it completely.
It is a line in robots.txt that says “wait X seconds before you ask for another page.”
02What to do about it
If your server struggles under heavy bot traffic, add a reasonable Crawl-delay (e.g., 5–10 seconds) to robots.txt and monitor server logs for a drop in request frequency. Combine it with User-agent: * to apply to all bots, or target specific agents like User-agent: Googlebot. After editing, test the file with the Google Search Console robots.txt tester to confirm syntax.
- Set a delay that balances crawl coverage with server capacity.
- Verify the change in Search Console before deploying site‑wide.
- Remove the directive once traffic normalises to avoid slowing legitimate indexing.
03How it is measured or noticed
Check server access logs for the time gap between consecutive requests from the same user‑agent. A consistent interval matching the declared delay indicates compliance. In Google Search Console, the Crawl Stats report shows average requests per day; a sudden drop after adding the directive suggests the bot respects it. Note that non‑compliant bots will show no change.
04Common mistakes
- Using a very high delay (e.g., 60 seconds) which can starve the index of fresh content.
- Placing the directive outside a
User-agentblock, causing parsers to ignore it. - Assuming all crawlers obey the rule; malicious scrapers typically disregard it.
05Limits and confusion
Crawl-delay is not part of the official Robots Exclusion Protocol (RFC 9309) and is unsupported by Bingbot, YandexBot, and many SEO tools. It also does not replace proper rate‑limiting at the server or CDN level. Do not confuse it with Request-rate (a non‑standard extension) or with the Crawl-Delay HTTP header, which does not exist.
06Worked example
User-agent: *
Crawl-delay: 5
Disallow: /private/
Frequently asked questions
How does Crawl Delay differ from the standard robots.txt Disallow directive?
Crawl Delay only suggests a pause between successive requests, while Disallow blocks access entirely. It does not prevent a bot from crawling, it merely spaces out the requests.
Should I add a Crawl Delay to my robots.txt if my server is under heavy bot traffic?
Yes, if you observe high request rates causing performance issues, a modest delay such as 5–10 seconds can reduce load. Monitor logs afterward to confirm the traffic drop.
How do I specify a Crawl Delay for a specific user‑agent in robots.txt?
Add a User‑agent line followed by a Crawl-delay line, for example User-agent: * Crawl-delay: 10 to request a ten‑second pause for all compliant bots.
Does Googlebot respect the Crawl Delay directive?
No, Googlebot ignores Crawl Delay and uses its own crawl‑rate settings in Search Console. Only some smaller bots honor the directive.
What happens if I set an excessively high Crawl Delay?
Search engines may crawl your site too slowly, delaying index updates and potentially hurting visibility. You would notice slower appearance of new content in search results.
How long after adding a Crawl Delay will I see a change in server logs?
The effect appears as soon as compliant bots re‑read robots.txt, typically within a few hours. Check the time gap between consecutive requests from the same user‑agent.
Asked out loud
spoken, not typedThe same term in the words somebody uses speaking to an assistant rather than typing into a box — written from the situation, which is why each one carries the situation it came from.
Yes, adding a Crawl-delay line to robots.txt will make compliant bots pause between requests, reducing load within a few hours.
Usually, you should verify whether a Crawl-delay is set too high or if bots ignore it, then adjust the value or use Google Search Console to manage crawl rate.
Usually, that line tells compliant crawlers to wait the specified seconds between requests, but major bots like Googlebot ignore it.