A standard used by webmasters to communicate restricted website access to automated crawlers.
Webmasters reading it alongside documentation regarding web crawling and indexing.
01What it is and how it works
When a crawler visits a site, it first requests https://example.com/robots.txt. The file contains directives grouped by user-agent. A User-agent line identifies which crawler the rules apply to — * means all. Below it, Disallow lists paths the crawler should not request, and Allow overrides a disallow for a specific path. Crawl-delay suggests a pause between requests. The protocol is defined in RFC 9309. It is a request, not a command: compliant crawlers obey, but non-compliant ones may ignore it. For example, a file with User-agent: * Disallow: /private/ tells every crawler to skip the /private/ directory.
REP is the rulebook that tells web crawlers where they are allowed to go on your site. You write a simple text file called robots.txt and put it in your site's root folder. Crawlers check this file first before fetching any page.
02What to do about it
Start by creating a robots.txt file and placing it in your domain's root directory (e.g., public_html/robots.txt). Use a plain text editor. List the crawlers you want to control. For AI crawlers like GPTBot or CCBot, add specific user-agent lines: User-agent: GPTBot Disallow: /. Test the file with Google's robots.txt tester in Search Console. Update the file whenever you add new sections you want to block or allow. Set a reasonable Crawl-delay if your server is slow. Monitor server logs to see if crawlers are hitting blocked paths — if they are, the crawler may not respect REP.
03How it is measured or noticed
You can check whether your robots.txt is working by visiting https://yourdomain.com/robots.txt in a browser — it should return a 200 status and display the file. Use Google Search Console's Crawl Stats report to see how often Googlebot requests pages and whether any are blocked. Look at server access logs for requests from known crawler IPs; if a crawler requests a disallowed URL, it is ignoring REP. Third-party tools like robots.txt validators can flag syntax errors. Also monitor your site's index coverage: if pages that should be blocked appear in search results, your REP may be misconfigured or the crawler may not support it.
04Common mistakes
- Blocking CSS or JavaScript files — this breaks page rendering and can hurt search rankings.
- Using incorrect syntax, like missing colons or extra spaces — crawlers may ignore the directive.
- Placing
robots.txtin a subdirectory instead of the root — crawlers will not find it. - Forgetting to update
robots.txtafter moving or deleting content — old disallows may block new pages. - Using
Disallow: /for all user-agents when only specific crawlers should be blocked — this blocks all crawlers, including Googlebot. - Relying on REP for security — it is not a security measure; sensitive data should be protected with authentication.
05Limits
REP is advisory. Malicious crawlers, spam bots, and some AI scrapers ignore robots.txt entirely. It controls crawling, not indexing: a page that is crawled and found may still appear in search results unless you also use a noindex meta tag or X-Robots-Tag HTTP header. REP applies only to the exact domain and protocol; a subdomain like blog.example.com needs its own robots.txt. It is often confused with a firewall or access control — it is neither. Some crawlers, especially those used for AI training, may not support the full REP standard; check their documentation. Finally, REP cannot block individual pages within a directory unless you list each path separately.
06Worked example
User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/
Crawl-delay: 10
User-agent: GPTBot
Disallow: /
User-agent: Googlebot
Allow: /
Frequently asked questions
What's the difference between robots.txt and meta robots tags?
They serve different purposes. robots.txt blocks crawlers from accessing pages, while meta robots tags control indexing and link following on individual pages. robots.txt is advisory; meta tags are directives that most crawlers respect.
Should I use robots.txt to block AI crawlers?
It depends on your goals. If you want to prevent AI models from training on your content, robots.txt can request that, but it's advisory and not all crawlers obey. You may need additional measures like terms of service or IP blocking.
How do I create a robots.txt file?
Create a plain text file named robots.txt and place it in your domain's root directory. Use directives like User-agent and Disallow to specify which crawlers and which paths to block. Example: User-agent: * Disallow: /private/.
Does robots.txt still work for modern search engines?
Yes, most reputable crawlers still check robots.txt, but it's advisory. Some AI crawlers may ignore it. It's a good first step but not a security measure.
What happens if I block the wrong paths in robots.txt?
You might accidentally block important pages from being indexed, harming SEO. For example, blocking CSS or JS files can cause rendering issues. Always test and use Allow directives carefully.
How long does it take for robots.txt changes to take effect?
Crawlers typically check robots.txt on each visit, so changes take effect immediately for new crawls. However, cached versions may persist for a few hours. You can use the URL Inspection tool in Google Search Console to verify.
Asked out loud
spoken, not typedThe same term in the words somebody uses speaking to an assistant rather than typing into a box — written from the situation, which is why each one carries the situation it came from.
Yes, use a robots.txt file with a Disallow rule for the test directory. That's the Robot Exclusion Protocol.
Yes, create a robots.txt file with a User-agent: * Disallow: / rule, but note it's advisory. That's the Robot Exclusion Protocol.
Edit your robots.txt file to remove the Disallow: / line or change it to Allow: /. That's the Robot Exclusion Protocol.