WordPress robots.txt Checker

1. The Robots Exclusion Protocol Standard

The Robots Exclusion Protocol is a voluntary standard — documented by Google Search Central — that well-behaved crawlers check before fetching anything else on a domain. A `robots.txt` file lists `User-agent` blocks, each with `Allow`/`Disallow` path rules and an optional `Sitemap:` pointer. Because compliance is voluntary, it only works against crawlers that respect it (all major search engines do); it is not a security or access-control mechanism, and disallowed URLs can still be discovered and even indexed by URL alone if linked from elsewhere.

2. Validating XML Sitemap References

A `Sitemap:` directive in robots.txt is the standard way to tell crawlers where your XML sitemap lives without requiring manual submission through each search engine's webmaster tools. If this check finds no sitemap directive, or you want to confirm the referenced sitemap is actually well-formed, run it through the WordPress XML Sitemap Validator next.

3. Dynamic vs Physical robots.txt in WordPress

WordPress generates a virtual `robots.txt` on the fly (via its `do_robots` hook) whenever no physical `robots.txt` file exists in the site's root directory — this default output disallows `/wp-admin/` while explicitly allowing `/wp-admin/admin-ajax.php`, and adds a Sitemap directive automatically if a sitemap-generating SEO plugin is active. The moment a physical `robots.txt` file is uploaded to the server root, WordPress's virtual version is completely bypassed and the static file takes over — which means a leftover `Disallow: /` from a staging environment's physical file will silently override whatever WordPress or an SEO plugin would otherwise have generated.

4. Instant Crawling via Nimbica Static Edge

A correct robots.txt tells crawlers what they're allowed to fetch — how fast those fetches complete is a separate factor that also affects how efficiently a site's crawl budget gets used. Nimbica pre-renders WordPress pages into static HTML served from edge locations, giving crawlers fast, consistent response times so the crawl budget robots.txt is protecting isn't also being wasted on slow origin server round-trips.

5. How to Use This Check

1. Run the scan and check the two flagged results first: the global disallow alert and the sitemap directive.
2. Review the raw file content for anything unexpected — commented-out rules, duplicate User-agent blocks, or paths that look like leftovers from a previous developer's staging setup.
3. If a physical robots.txt exists and you're not sure why, check whether it's actually needed — for many WordPress sites, removing it and relying on WordPress's own virtual robots.txt (managed automatically by an SEO plugin) is simpler and less error-prone.
4. Also check core version exposure with the WordPress Version & Update Checker — robots.txt hardening and version exposure are both quick wins worth checking together during a basic site audit.

6. Common Mistakes

• Forgetting to remove a staging "discourage search engines" robots.txt after launch. This is the single most common cause of an accidental full-site block — always re-check robots.txt immediately after any staging-to-production migration.
• Using robots.txt to try to hide sensitive content. Disallowed URLs are still publicly listed in the robots.txt file itself and can still be discovered and linked to from elsewhere — robots.txt is a crawl-budget tool, not an access-control mechanism.
• Blocking admin-ajax.php along with the rest of /wp-admin/. Some overly broad robots.txt rules accidentally disallow this specific file, which can break front-end functionality that depends on it for legitimate AJAX calls.

7. Who Should Use This Tool

Anyone who just launched or migrated a WordPress site (to catch a leftover staging block before it costs weeks of lost indexing), agencies doing a technical SEO audit, and site owners who want a quick sanity check that their sitemap is actually being advertised to crawlers correctly.

8. Limitations

This check fetches the live file and flags the two most common, highest-impact WordPress robots.txt problems — it is not a full syntax validator against the entire Robots Exclusion Protocol spec (wildcard patterns, crawl-delay, per-user-agent rule groups aren't individually validated). A clean result here means these two specific checks passed, not that every rule in the file is optimally configured.