Missing Robots.txt

What This Means

The website does not have a robots.txt file. While not strictly required, a robots.txt file helps communicate crawling preferences to search engines and AI crawlers, and is considered a best practice for SEO.

What Triggers This Issue

This issue is triggered when a request to /robots.txt returns a 404 Not Found status code, indicating no robots.txt file exists at the domain root.

How To Fix

Create a robots.txt file at your domain root (e.g., https://example.com/robots.txt ). At minimum, include a Sitemap directive pointing to your XML sitemap. Consider adding rules for different user agents based on your crawling preferences.

← Back to Robots.txt