robots.txt Generator
Generate a crawl-optimized robots.txt file for your WordPress site, WooCommerce store, or custom web application in seconds.
Declares your XML sitemap location directly to Googlebot and Bingbot.
WordPress & Directory Protection
AI Scrapers & Bot Protection
Custom Directives
# Generated by Nimbica Free Tools (https://www.nimbica.com/tool/robots-txt-generator)
# Optimized for WordPress, Technical SEO, and Server Performance
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /xmlrpc.php
Allow: /wp-admin/admin-ajax.php
# Block AI Scrapers & Data Mining Bots
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Anthropic-ai
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: ByteSpider
Disallow: /
User-agent: Diffbot
Disallow: /
Sitemap: https://example.com/sitemap.xmlhttps://yourdomain.com/robots.txt.How to Generate & Deploy Your robots.txt File
Select a Preset
Choose the WordPress Default or WooCommerce preset to load optimized rules automatically.
Configure Rules
Enter your XML sitemap URL, toggle AI scraper protection, and add custom Allow or Disallow paths.
Copy or Download
Click Copy Code or Download to save the generated plain text `robots.txt` output.
Upload to Site Root
Place the file in your root folder (e.g. `public_html/robots.txt`) via FTP, cPanel, or your hosting control panel.
Complete Guide to WordPress robots.txt Optimization
The robots.txt file is one of the most fundamental technical SEO assets of any website. It serves as an instruction manual for automated web crawlers like Googlebot, Bingbot, DuckDuckGo, and social media scrapers, establishing clear boundaries on which parts of your web server are open for inspection and which paths should be ignored.
1. robots.txt vs. Meta Noindex Tag: When to Use Which?
A frequent confusion among website owners is mistaking robots.txt for an indexation control tool. While both influence search engines, they operate at completely different stages of the search pipeline:
| Feature | robots.txt Directives | Meta Noindex Tag |
|---|---|---|
| Primary Goal | Prevents crawlers from requesting page bandwidth | Prevents page from appearing in Google search results |
| Googlebot Action | Does not download or execute the page HTML | Fetches the page HTML and reads the noindex directive |
| Best Used For | Backend admin folders (`/wp-admin/`), internal search, API files | Thank-you pages, staging URLs, thin landing pages, checkout |
| Crawl Budget Impact | Saves server CPU & crawl quota instantly | Consumes crawl budget on initial fetch |
Rule of Thumb: If you want to stop bots from hitting backend files or consuming PHP resources, use robots.txt. If a page is publicly reachable but you simply don’t want it in Google Search, use a noindex meta tag or pair it with structured schema markup.
2. Common robots.txt Mistakes That Hurt WordPress SEO
A single syntax error in your robots.txt file can cause Google to drop your pages from search results or render your layout incorrectly. Here are the top 4 mistakes to avoid:
- Blocking CSS and JavaScript Assets: Historically, site owners disallowed
/wp-includes/js/or/wp-content/themes/. Modern Googlebot renders pages like a real browser. If you block CSS/JS files, Google cannot verify mobile responsiveness or layout stability, hurting your Core Web Vitals scores. - Accidental Stray Slash (`Disallow: /`): Adding a single slash under
User-agent: *blocks your entire website from Google. Always verify your disallow directives before publishing. - Case Sensitivity Errors: Directives in robots.txt are case-sensitive. Disallowing
/Category/will not block/category/. - Forgetting the XML Sitemap Reference: Including your absolute sitemap URL at the bottom of your file allows Googlebot to locate new posts instantly.
3. Crawl Budget Explained: Why Dynamic WordPress Sites Suffer
Crawl Budget is the number of URLs Googlebot can and wants to crawl on your site during a given timeframe. On traditional dynamic WordPress servers, every Googlebot request triggers PHP execution, MySQL database queries, and plugin overhead. If your server is slow (high TTFB), Googlebot reduces its crawl rate to avoid crashing your host.
By filtering out unnecessary backend requests using robots.txt and converting your site to a static architecture via Nimbica, every page request is served from CDN edge cache in under 45ms. This allows Googlebot to index 10x more pages per second without any server load.
4. How Googlebot Actually Reads & Caches robots.txt
According to Google Search Central’s official documentation, Googlebot fetches robots.txt before visiting any page on your domain. Google caches the file for up to 24 hours.
If your server returns a 200 OK with valid rules, Googlebot follows them strictly. If your server returns a 404 Not Found, Googlebot assumes there are no restrictions and crawls all public URLs. However, if your host returns a 5xx Server Error, Googlebot treats it as a temporary site failure and halts crawling until the file is accessible.
Tired of managing WordPress server crawler crashes?
Nimbica converts your WordPress website into a pre-rendered static site served from 300+ global Cloudflare edge locations — delivering sub-45ms speed and eliminating PHP server crashes during bot traffic spikes.
Frequently Asked Questions
What is a robots.txt file and why do I need one?
A robots.txt file is a plain text file placed in your website root directory that tells search engine crawlers (like Googlebot and Bingbot) which pages or files they can or cannot request. It helps manage your crawl budget and protects internal backend scripts from being wasted on unnecessary crawls.
Where should I upload the generated robots.txt file?
Upload the robots.txt file directly to your website root public directory (e.g., https://yourdomain.com/robots.txt). For WordPress users, you can upload via FTP/cPanel or paste the code into your SEO plugin setting page.
Why should I block AI scrapers in my robots.txt file?
AI scrapers (such as GPTBot, CCBot, and PerplexityBot) continuously crawl websites to train LLMs and generate AI answers. Blocking them protects your original copyrighted content and prevents automated bots from consuming valuable server bandwidth.
Does blocking a path in robots.txt remove it from Google search results?
Not automatically. Robots.txt prevents crawlers from visiting the page, but if another site links to that URL, Google may still index the URL without reading its content. To completely remove a page from search results, use a meta noindex tag instead.
Is this robots.txt Generator completely free to use?
Yes, 100% free with no sign-up, email requirement, or usage limits.
Related Technical SEO Utilities
Schema Markup Generator
Generate valid JSON-LD structured data for FAQ, Product, and HowTo Rich Snippets.
Open Graph Preview Tool
Preview how your web page looks when shared on Twitter/X, Facebook, and LinkedIn.
WordPress Speed Test
Test your WordPress site load time and receive Core Web Vitals diagnostic recommendations.