Robots.txt
This section covers all robots.txt related issues that can be detected during a website audit.
Issues
- AI Crawlers Blocked
- All Crawlers Blocked
- Bingbot Blocked
- Googlebot Blocked
- Invalid Sitemap URL in Robots.txt
- Missing Robots.txt
- No Sitemap Directive
- Robots.txt Server Error
- Robots.txt Syntax Errors
Last updated on