The number of requests a search engine makes to a website per second.
Webmasters reading about site performance, indexing strategies, and server health.
01What it is and how it works
Googlebot uses a heuristic algorithm to decide the crawl rate for each site. That algorithm looks at your server's capacity, how fast it responds, and whether it returns errors. If your server is fast and healthy, Googlebot increases the rate. If it sees slow responses or failures, it backs off. The crawl rate is dynamic, not a fixed number. You can influence it indirectly by keeping your server responsive and your site error-free. Google ignores the crawl-delay directive in robots.txt, though other search engines may honor it.
Crawl rate is how fast a search engine visits your site. It's the frequency of requests.
02What to do about it
To manage your crawl rate, start with server performance. Open Google Search Console's Crawl Stats report to see daily requests and average response time. If you notice a drop, check for server errors or recent robots.txt changes. To encourage a higher crawl rate, make sure your server can handle more concurrent requests. Use the URL Inspection tool to request indexing for important pages. Avoid blocking Googlebot with robots.txt unless you have a strong reason. Keep your sitemap current and remove dead links.
03How it is measured or noticed
You can measure crawl rate by examining your server logs. Filter for Googlebot's user agent and count requests per second or per hour. Google Search Console's Crawl Stats report shows requests per day, average response time, and download size. A sudden change in crawl rate often signals a problem, such as a server outage or a robots.txt mistake. Real-time analytics tools can also show crawl traffic as it happens.
04Common mistakes
- Blocking crawlers with robots.txt, which prevents Google from indexing your site.
- Setting a low crawl-delay for Google, which is ignored and wastes time.
- Ignoring crawl errors in Search Console, which can lower your crawl rate.
- Using robots.txt to block CSS or JavaScript, which hurts rendering and may reduce crawl rate.
- Assuming a high crawl rate is always good; it can overload your server.
05Limits and confusions
Crawl rate is not the same as crawl budget. Crawl budget is the total number of URLs Google can crawl in a given period, while crawl rate is the speed of requests. You cannot directly set crawl rate; it's determined by Google's algorithm. It doesn't apply to all search engines—Bing and others have their own systems. Crawl rate is not a ranking factor. A site with a low crawl rate can still rank well if its content is relevant and authoritative.
06Worked example
Consider a site with 1,000 pages. If Googlebot crawls at 5 requests per second, it would take 200 seconds to crawl all pages. But if the server starts responding slowly, Googlebot reduces the rate to 1 request per second, extending the crawl to 1,000 seconds. This shows how server performance directly affects crawl rate.
Frequently asked questions
How is crawl rate different from crawl budget?
Crawl rate is the speed at which a search engine requests pages, while crawl budget is the total number of URLs a search engine will crawl on your site within a given timeframe. Your crawl rate directly affects your crawl budget. Think of crawl rate as the pace and crawl budget as the overall allowance.
How do I measure my crawl rate?
You can measure crawl rate by examining your server logs to count requests from Googlebot or other search engine bots per second or per minute. Tools like Google Search Console provide crawl stats. This shows you the actual rate and which pages are being crawled.
When should I adjust my crawl rate?
You should adjust crawl rate when your server performance suffers from the load. Start by checking server performance and response times. If you're getting errors or slow responses, you can lower the crawl rate in Google Search Console.
What happens if my crawl rate is too high?
If your crawl rate is too high, your server may slow down or even crash, causing downtime and performance issues for real users. You might notice 5xx errors in logs. This can hurt your site's user experience and indirectly affect rankings.
Asked out loud
spoken, not typedThe same term in the words somebody uses speaking to an assistant rather than typing into a box — written from the situation, which is why each one carries the situation it came from.
Yes, it could be. Check your server logs for a spike in Googlebot requests. If the crawl rate is unusually high, you can lower it in Google Search Console to ease the load.
It depends on your site's size and popularity. For larger sites, many requests are normal, but if you're seeing errors or slowdowns, you might need to adjust your crawl rate.
Yes, you can set a custom crawl rate in Google Search Console without affecting rankings. Lowering it just slows how quickly Google discovers and re-crawls your content, not how it ranks it.