X-RAY // WEB // ROBOTS

ROBOTS.TXT
CHECK.

Read the robots.txt of the site, find a forgotten Disallow: /, the declared sitemaps and the rules for GPTBot, ClaudeBot and other AI crawlers.

Runs only this check. For all 33 checks at once, use the full Domain X-Ray. Public lookups only; nothing is stored on the server.

ROBOTS

robots.txt

WAITING

Enter a domain above to run this check.

01 // WHAT THE CHECK VERIFIES

Reads /robots.txt: the rules for search engines and AI robots, and the declared sitemaps.

02 // WHY IT MATTERS

A forgotten Disallow: / takes the whole site out of Google. AI robots read these rules too.

03 // HOW TO READ THE RESULT

OK: robots.txt published, without a general block for every robot.
WARNING: The /robots.txt address returns an HTML page or another unexpected code.
FAIL: Disallow: / for every robot, or /robots.txt answers 5xx, and then Google stops crawling the site.
INFO: No robots.txt (404): every robot may crawl everything, which is usually what you want.

ERROR and TIMEOUT mean the query did not finish (the source was down or slow), not a problem with the domain: run the check again.

04 // HOW TO FIX

Allow crawling and declare the sitemap:

User-agent: *
Disallow:

Sitemap: https://example.com/sitemap.xml

Would rather have someone else do it? Bit Happens can fix this →

05 // CONTINUE