Robots.txt

Robots.txt is a plain-text file at the root of a website that tells crawlers which parts of the site they may or may not fetch — an editorial signal, not a security control.

Also known as: robots file, crawler directives, robots exclusion

Robots.txt is a plain-text file at the root of a website (example.com/robots.txt) that tells crawlers which parts of the site they may or may not fetch. It uses a simple syntax of user-agent declarations and allow/disallow rules. Robots.txt is one of the oldest web standards, defined by the Robots Exclusion Protocol, and remains the primary mechanism for managing crawler access at scale.

What Robots.txt Means

Robots.txt is a plain-text file at the root of a website (example.com/robots.txt) that tells crawlers which parts of the site they may or may not fetch. It uses a simple syntax of user-agent declarations and allow/disallow rules. The file is fetched by crawlers as the first request when visiting a domain, before crawling any other pages. Path-based rules can block specific URL patterns: Disallow: /admin/ blocks anything under /admin/; Disallow: /*.pdf$ blocks all PDF files. Wildcards (*) and end-anchors ($) are supported by most major crawlers. The file must be served with a 200 response and plain-text content type.

How Robots.txt Works

Robots.txt works as an editorial signal, not a security control. Reputable search and AI crawlers — Googlebot, Bingbot, GPTBot, ClaudeBot, and most others — honor robots.txt directives, but compliance is voluntary. Less reputable bots and direct URL access ignore it. Anything genuinely confidential must be protected by authentication, not by a disallow rule. The right mental model is ‘request,’ not ‘enforce.’ Disallow prevents crawling — bots won’t fetch the URL. Noindex (in meta tags or HTTP headers) prevents indexing — bots may crawl but won’t add the URL to the index. They aren’t interchangeable, and using disallow alone can leave URLs visible in search with sparse listings.

Common Pitfalls and Misconceptions

A common mistake is conflating disallow in robots.txt with noindex. Robots.txt prevents crawling; noindex (in meta tags or HTTP headers) prevents indexing. A page disallowed in robots.txt can still appear in search results if other signals (external links, sitemap entries) make Google aware of the URL — Google just won’t have crawled the page, so it shows a sparse listing without snippet. To reliably keep a page out of search, allow crawl and use noindex. Another error is listing sensitive paths in robots.txt — the file is publicly readable, so disallowing /admin/ effectively advertises its existence to anyone curious enough to check.

Robots.txt in Practice

The practitioner pattern is to treat robots.txt as a deliberate, version-controlled policy file rather than ad hoc rules accumulated over years. Audit periodically against site architecture and crawler behavior: which user agents are addressed, which paths are blocked, whether any important sections have been accidentally disallowed, and whether the file accurately reflects current intent. AI crawler additions over the last few years have made many older robots.txt files incomplete — addressing GPTBot, ClaudeBot, Google-Extended, and the rest is now part of routine maintenance, not an exotic addition. Test changes in Google Search Console’s robots.txt tester before deploying.

Back to the Glossary

Common questions.

What is robots.txt used for?

It tells web crawlers which parts of a site they may or may not fetch, using a simple syntax of user-agent declarations and allow/disallow rules. Reputable search and AI crawlers honor it; less reputable bots may ignore it. Robots.txt is an editorial signal, not a security mechanism — anything confidential needs proper authentication.

Where does robots.txt live and how is it found?

At the root of the domain, accessed at example.com/robots.txt. Crawlers fetch it as the first request when visiting a domain, before crawling any other pages. The file must be served with a 200 response code and plain-text content type; misconfigurations can cause crawlers to either crawl everything or block everything by mistake.

What's the difference between disallow in robots.txt and noindex?

Disallow prevents crawling — bots won't fetch the URL. Noindex (in a meta tag or HTTP header) prevents indexing — bots may crawl the URL but won't add it to the search index. They're not interchangeable. To reliably keep a page out of search results, allow crawl and use noindex; using disallow alone can leave the URL visible in search with a sparse listing.

How do you address AI crawlers in robots.txt?

Add explicit user-agent rules for the AI crawlers you want to allow or block: GPTBot (OpenAI training), OAI-SearchBot (ChatGPT search), ClaudeBot (Anthropic), Google-Extended (Google AI training, separate from Googlebot), PerplexityBot, and others. Each can be allowed or disallowed independently. The list of relevant AI crawlers evolves; review it quarterly.

Can robots.txt block specific URL patterns?

Yes, using path-based rules. Disallow: /admin/ blocks anything under /admin/. Disallow: /*.pdf$ blocks all PDF files. Wildcards (*) and end-anchors ($) are supported by most major crawlers. The syntax is simple but has edge cases; test rules in Google Search Console's robots.txt tester before deploying to production.

What happens when robots.txt has errors?

Depends on the error. A 5xx response code typically causes Googlebot to halt all crawling temporarily until robots.txt returns successfully. A 404 means there's no robots.txt and bots crawl everything. Syntax errors are usually ignored on a per-line basis but can leave intended blocks unenforced. Test changes before deploying, and monitor crawl behavior afterward.

Should sensitive pages be blocked in robots.txt?

No. Robots.txt is publicly readable, so listing sensitive paths there advertises their existence. And it doesn't actually secure anything — direct URL access still works. Use authentication for genuinely sensitive content. Robots.txt is appropriate for managing crawl behavior, not for protecting confidential material from determined access.

Related Terms

Let’s talk about what your next quarter could look like.

Tell us what you’re working on. A senior practitioner reads it, not an SDR queue, and replies, usually within one business day.

Reviewed personally, not routed through a queue.
A conversation about what you’re actually working on, not a generic pitch.
No pressure, just a chance to talk it through.