Crawl & SEO
Robots.txt Checker
Read crawler directives and sitemap declarations in robots.txt.
When to use this robots.txt checker
Read robots.txt when checking crawler restrictions or sitemap declarations after a launch. The file is fetched at the origin, not relative to an individual page's path.
What to look for
- Look for the relevant User-agent group and Allow or Disallow rules.
- Read declared sitemap locations; they may differ from /sitemap.xml.
- Check page indexing directives separately and use Search Console for Google's actual crawl observations.
How to read the results
Fetches /robots.txt at the entered origin. Directives are reported, not simulated for a particular crawler and path.
PASS marks a stated check that passed. WARNING deserves review. FAIL marks a failed check. INFORMATION reports a fact. Results are a snapshot from our server, not a score or certification.
Does robots.txt prevent a URL appearing in search?
Blocking crawling does not reliably prevent indexing. A URL may be known through links. A noindex directive needs to be accessible to the crawler to be read; robots.txt is not access control.
Download your results as Markdown to save the observations or discuss them with your AI assistant. Reports include the check timestamp and limitations.