What is robots.txt?
robots.txt is a file at a site’s root that tells automated crawlers which paths they may request. It is advisory rather than enforced, applies per user-agent, and is the file that determines whether AI crawlers such as GPTBot, ClaudeBot and PerplexityBot are permitted to read a site at all.
- Category
- Search, AEO & AI Visibility
- Also called
- Robots exclusion protocol
The most common self-inflicted AI-visibility problem lives here: a robots.txt copied from another project, or a blanket disallow added during development, that quietly blocks the crawlers behind the answer engines. A site cannot be cited by a system that was never allowed to read it.
Blocking is a legitimate choice — some publishers deliberately exclude AI training crawlers — but it should be a decision rather than an accident, and it is worth distinguishing the crawlers used for training from those used to fetch pages at answer time.
Related terms.
llms.txt
Search, AEO & AI Visibilityllms.txt is a proposed plain-text file at a site’s root that offers language models a curated map of its most useful pages, in Markdown, without navigation and boilerplate.
Generative Engine Optimization (GEO)
Search, AEO & AI VisibilityGenerative engine optimization is the practice of increasing how often a brand or page is used as a source by generative AI systems.
XML Sitemap
Search, AEO & AI VisibilityAn XML sitemap is a machine-readable list of the URLs a site wants crawled, usually with a last-modified date for each.
Scoping a project?
Reading definitions because you are scoping a project? Skip ahead and just ask.