robots.txt Validator
Scan a live site's robots.txt or paste custom directive text to check for a blanket Disallow: / block and a missing sitemap declaration.
{your-domain}/robots.txt with a single read-only GET request and checks it for two things — a blanket Disallow: / block and a Sitemap: declaration. It is not a full syntax validator and does not inspect anything beyond the robots.txt file itself.Robots.txt Specification & Crawl Budget Management
1. Google Official Robots.txt Specification
Robots.txt manages crawler access to your web server. As documented in Google Search Central Robots.txt Specification Guide, valid directive formatting prevents accidental site de-indexing.
2. Common Robots.txt Syntax Errors & Solutions
Avoid using wildcard `*` incorrectly in `Disallow` paths which can block entire directories.
3. Auditing XML Sitemap Health
Validate XML sitemap structure in our XML Sitemap Validator.
4. Instant Crawling via Nimbica Static Edge
Nimbica pre-renders robots.txt files into static assets served from 300+ global edge locations, ensuring search bots fetch directives with sub-50ms TTFB.
5. How to Use This Validator
To check a live site, enter its domain in Option 1 and click “Scan URL” — the server fetches /robots.txt from that domain and reports the HTTP status, line count, whether a blanket disallow block exists, and whether a sitemap is declared, along with the raw file contents. To check draft or custom directive text before publishing, paste it into Option 2's textarea; the same two checks run instantly client-side as you type, with no request sent anywhere.
Use Option 1 to audit a site that is already live (yours or a competitor's public robots.txt), and Option 2 to sanity-check a new robots.txt file before you deploy it — catching an accidental Disallow: / before it goes live is far cheaper than catching it after Google stops crawling your site.
6. Who Should Use This Tool
This is useful for developers deploying a new site or staging environment (where an accidentally-inherited Disallow: / is a common and easy-to-miss mistake), SEOs doing a quick health check on a client's crawl directives, and anyone auditing whether their sitemap is properly referenced for crawler discovery. It complements a full technical SEO audit rather than replacing one.
7. Limitations
This tool checks for exactly two conditions — a blanket disallow block and a sitemap declaration — and does not validate general robots.txt syntax, catch malformed directive lines, verify per-user-agent blocking rules, or check wildcard/path-matching correctness beyond those two patterns. A robots.txt file can pass both checks here and still contain other mistakes (blocking a specific important directory, conflicting Allow/Disallow rules, or an incorrectly scoped user-agent group) that this tool will not detect. For a directive-by-directive breakdown, use Google Search Console's robots.txt report against a verified property, which reflects exactly how Googlebot interprets your file.
Give search engine crawlers sub-50ms response times with Nimbica
Transform dynamic PHP rendering bottlenecks into ultra-fast static HTML deployed across 300+ global edge locations.
Frequently Asked Questions
What does a robots.txt file do?
A robots.txt file instructs search engine crawlers (like Googlebot) which URLs or directories they can or cannot request from your server. It is a crawling directive, not an indexing directive — a disallowed page can sometimes still appear in search results if other sites link to it.
Why is Disallow: / dangerous in robots.txt?
`Disallow: /` under a wildcard `User-agent: *` block tells every crawler not to request any URL on your site, which can prevent your pages from being crawled and ranked in Google Search. It is a common accidental mistake — often left over from a staging environment robots.txt that got deployed to production.
Should XML sitemap URLs be declared in robots.txt?
Yes, it is recommended. Declaring `Sitemap: https://example.com/sitemap.xml` in robots.txt helps crawlers discover your sitemap, though submitting it directly in Google Search Console works independently and does not require the robots.txt reference.
What exactly does this tool check — is it a full syntax validator?
No. This tool checks two specific things: whether a blanket `Disallow: /` block exists, and whether a `Sitemap:` line is present. It does not perform full robots.txt syntax validation — it will not catch malformed directive lines, incorrect user-agent groupings, or invalid wildcard patterns beyond those two checks. For directive-by-directive validation, use Google Search Console's robots.txt report on a verified property.
Does scanning a live URL modify or affect my website in any way?
No. Scanning a live URL performs a single read-only HTTP GET request to fetch `/robots.txt` from the domain you enter — nothing is written, changed, or stored on your site, and no authentication or write access is required.
Related Tools in This Cluster
XML Sitemap Validator
Validate XML sitemap structure, urlset declarations, and HTTP response health.
Title Tag Length Checker
Audit pixel width and character length of your meta titles to prevent SERP truncation.
Meta Tag Generator
Generate complete HTML meta tags, Open Graph social tags, and Twitter Card tags.