Robots.txt Tester
Validate your robots.txt syntax and test URL access for different search engine and AI crawlers. Supports 20+ bots including Googlebot, GPTBot, Claude-Web, and more.
robots.txt
1
Enter robots.txt content to validate
How to Use This Tool
Paste your robots.txt content in the editor on the left, or load one of the preset templates. The tool will automatically validate the syntax and show any errors or warnings.
Features
- Syntax Validation: Checks for proper formatting and valid directives
- URL Testing: Test if specific URLs are allowed or blocked
- Bot Testing: See access status for 20+ search engines and AI crawlers
- Templates: Quick-start templates for common configurations
- Local Storage: Your content is saved automatically in your browser
About robots.txt
The robots.txt file tells search engine crawlers which URLs they can access on your site.
It's placed at the root of your domain (e.g., https://example.com/robots.txt) and uses
simple directives like User-agent, Allow, and Disallow.
AI Crawlers
Many AI companies use web crawlers to gather training data. Common AI crawlers include:
- GPTBot: OpenAI's crawler for ChatGPT training
- Claude-Web: Anthropic's crawler for Claude
- Google-Extended: Google's AI training crawler (separate from Googlebot)
- CCBot: Common Crawl's crawler used by many AI companies