robots.txt
robots.txt is a file at the site root that specifies which crawlers may fetch which parts of the site.
AI crawlers have their own names and can be allowed or blocked individually. A change can take time to take effect.
Related terms
Guides covering this term
All terms
Back to glossary