Robots.txt Checker
Fetch and inspect a site’s robots.txt file.
Read the guideRun the tool
JVR Software’s server processes the public target you enter. The diagnostic is rate-limited. Use a system you own or are authorised to check.
More information
Back to the toolWhat it is and when to use it
Robots.txt Checker displays the public crawler instructions at a website's root. Use it when a section is missing from search, after publishing a sitemap, or when checking that a staging rule was not copied onto your production website.
How it works
Given a public URL, the tool rebuilds the origin and requests /robots.txt at that origin. It returns the HTTP status, content type and a bounded text preview. Robots files use groups such as User-agent followed by Allow and Disallow paths. Sitemap lines identify XML sitemap URLs.
How to use it
- Paste the full site URL with the correct scheme and hostname.
- Look for the relevant crawler group and the rules that apply to your path.
- Open any listed sitemap separately or check it in your search console.
Worked example
If a public site returns a text file containing User-agent: * and Disallow: /, that group instructs compliant crawlers not to fetch the whole site. If your new public page should be crawlable, review the deployed rule in context and verify the rule that applies specifically to that page.
Reading the result
A 200 with plain text is a useful starting point. A 200 containing your home page's HTML may mean the server fallback is answering instead of a robots file. A 404 and a temporary server error are different cases that crawlers can handle differently.
If a public resource has vanished from search, read its own HTTP status, canonical and robots metadata too. An exclusion file governs crawler requests. Whether a crawler can discover an address or include it in an index also depends on signals elsewhere.
Common mistakes and limits
Robots rules are not access control. They do not stop people opening URLs and do not reliably remove already known URLs from search. Keep private content behind authentication and use the relevant indexing controls for pages that are publicly reachable but should not appear in results.
