- Key Takeaways<\/a>
- What Blocked by robots.txt Actually Means<\/a>
- Blocked by robots.txt vs Indexed, though blocked by robots.txt<\/a>
- Common WordPress-Specific Causes<\/a>
- The \u201cDiscourage Search Engines\u201d Setting<\/a>
- A Staging Site's Robots.txt Copied to Production<\/a>
- Security Plugins or Firewalls Adding Rules<\/a>
- Hosting Provider Virtual Robots.txt Files<\/a>
- SEO Plugin Misconfiguration<\/a><\/li><\/ul><\/li>
- Step by Step Fix<\/a>
- Step 1: Confirm the Issue in Google Search Console<\/a>
- Step 2: Check the WordPress Reading Settings First<\/a>
- Step 3: View Your Live Robots.txt File Directly<\/a>
- Step 4: Use the URL Inspection Tool for a Specific Page<\/a>
- Step 5: Edit the Robots.txt File<\/a>
- Step 6: Request Indexing to Confirm the Fix<\/a>
- Step 7: Monitor Search Console Over the Following Days<\/a><\/li><\/ul><\/li><\/ul><\/li>
- Quick Diagnostic Checklist<\/a>
- How to Prevent This From Happening Again<\/a>
- Frequently Asked Questions<\/a>
- Conclusion<\/a><\/ul><\/ul><\/ol>\t\t\t\t\t<\/div>\n\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\n\n\n
<\/p>\n\n\n\n
This guide walks through exactly what this error means, the specific WordPress causes behind it, and the steps to fix and confirm it, so your pages actually make it into Google’s index.<\/p>\n\n\n\n
Key Takeaways<\/h2>\n\n\n\n
- \n
- \u201cBlocked by robots.txt\u201d means Googlebot found a URL but was told not to crawl it, which is different from a page simply not being indexed yet.<\/li>\n\n\n\n
- On WordPress specifically, the most common cause is the built-in \u201cDiscourage search engines from indexing this site\u201d setting being left checked after launch.<\/li>\n\n\n\n
- A staging site’s robots.txt file accidentally carried over to the live site is another frequent, easy-to-miss cause.<\/li>\n\n\n\n
- Security plugins, firewalls, and some hosting providers can add their own Disallow rules without clearly telling you.<\/li>\n\n\n\n
- \u201cBlocked by robots.txt\u201d and \u201cIndexed, though blocked by robots.txt\u201d are two different statuses with two different meanings, and mixing them up leads to the wrong fix.<\/li>\n\n\n\n
- After fixing the issue, request indexing through Search Console rather than assuming Google will notice the change immediately on its own.<\/li>\n<\/ul>\n\n\n\n
What Blocked by robots.txt Actually Means<\/h2>\n\n\n\n
Every website can include a robots.txt file, a small, plain text file sitting at the root of your domain, that gives instructions to search engine crawlers about which parts of the site they should and should not access. When Google Search Console shows \u201cBlocked by robots.txt,\u201d it means Googlebot found a link to that URL somewhere, but your robots.txt file specifically told it not to crawl the page, so it never actually visited or read the content.<\/p>\n\n\n\n
This matters because robots.txt controls crawling, not indexing directly. A blocked page usually will not appear in search results, since Google has no content to evaluate, but in some cases a page can still get indexed anyway if enough other sites link to it, appearing with just a bare URL and no real description. That second scenario has its own separate status in Search Console, which is worth understanding clearly before you start making changes.<\/p>\n\n\n
\n![\"WordPress](\"https:\/\/digiinte.com\/blog\/wp-content\/uploads\/2026\/08\/What-Blocked-by-robots.txt-1024x576.webp\")
<\/figure>\n<\/div>\n\n\n<\/p>\n\n\n\n
Blocked by robots.txt vs Indexed, though blocked by robots.txt<\/h2>\n\n\n\n
These two statuses look similar but call for different fixes, and mixing them up is one of the most common mistakes people make when troubleshooting this issue.<\/p>\n\n\n\n
Blocked by robots.txt <\/strong>means Google has not crawled the page at all, and it is not indexed. If this is a page you want ranking, the fix is straightforward, remove the block.<\/p>\n\n\n\n
Indexed, though blocked by robots.txt <\/strong>means Google indexed the URL anyway, usually because other pages link to it, even though your robots.txt told it not to crawl the content itself. This status is more of a judgment call. If you genuinely wanted this page hidden, robots.txt alone is not enough, since it does not remove pages already indexed through outside links. If you want it properly indexed with a real title and description, you need to remove the block so Google can actually crawl and read it.<\/p>\n\n\n\n
Common WordPress-Specific Causes<\/h2>\n\n\n\n
General robots.txt advice often misses the causes that are specific to how WordPress sites get built, launched, and maintained.<\/p>\n\n\n\n
The \u201cDiscourage Search Engines\u201d Setting<\/h3>\n\n\n\n
WordPress includes a built-in setting under Settings, Reading called \u201cDiscourage search engines from indexing this site.\u201d It exists specifically for sites still under development, and when checked, WordPress automatically adds a Disallow rule to your virtual robots.txt file. <\/p>\n\n\n\n
The most common way sites end up blocked is simple, this setting gets enabled during development and never gets unchecked at launch. Developers building a site for a client often check this box deliberately while work is in progress, then hand the finished site over without walking through a final pre-launch checklist together, and the setting quietly stays on for weeks or months after the site technically goes live.<\/p>\n\n\n\n
Example: <\/strong>A small business owner launches a new WordPress site built by a freelancer, only to discover three months later that the site has never appeared in Google search results at all, traced back to this exact setting still being checked from the development phase.<\/p>\n\n\n
\n![\"WordPress](\"https:\/\/digiinte.com\/blog\/wp-content\/uploads\/2026\/08\/Discourage-Search-Engines-1024x683.webp\")
<\/figure>\n<\/div>\n\n\n<\/p>\n\n\n\n
A Staging Site’s Robots.txt Copied to Production<\/h3>\n\n\n\n
Development and staging environments are often deliberately blocked from search engines entirely, which is correct behavior for a site not meant to be public yet. Problems start when a site migration or deployment process copies that staging robots.txt file directly to the live production site, carrying the block over without anyone noticing until traffic mysteriously stops growing. <\/p>\n\n\n\n
This is especially common with automated deployment tools or one-click staging to production migrations, where every file, including one that should never have made the trip, gets copied over exactly as is.<\/p>\n\n\n\n
Example: <\/strong>An agency migrates a client site from a staging subdomain to the live domain using a full site copy tool, and the staging environment’s robots.txt file, which intentionally blocked all crawlers, comes along for the ride and silently blocks the newly launched live site too.<\/p>\n\n\n\n
Security Plugins or Firewalls Adding Rules<\/h3>\n\n\n\n
Some security plugins and firewall services modify robots.txt as part of their protective measures, sometimes blocking specific bot patterns or entire directories more broadly than intended. If you recently installed or reconfigured a security plugin around the time indexing issues started, this is worth checking directly. <\/p>\n\n\n\n
These tools often mean well, aggressively blocking anything that looks like a bot to prevent scraping or malicious crawling, but an overly broad rule can accidentally catch Googlebot in the same net meant for genuinely harmful traffic. Our guide on WordPress security plugins<\/a> covers what these tools typically do, useful context if you suspect one is the culprit here.<\/p>\n\n\n\n
Hosting Provider Virtual Robots.txt Files<\/h3>\n\n\n\n
Some hosting environments generate their own virtual robots.txt file at the server level, separate from anything WordPress or your plugins control directly. This can create confusing situations where the robots.txt file you edit through WordPress does not match what Googlebot actually receives, since the server-level version takes precedence. This is more common with certain managed WordPress hosts that apply their own default configurations, particularly for staging environments or password-protected development sites, and the setting can sometimes persist even after a site is fully launched and password protection is removed.<\/p>\n\n\n\n
SEO Plugin Misconfiguration<\/h3>\n\n\n\n
Plugins like Yoast SEO and Rank Math both include their own robots.txt editing tools and site-wide indexing controls. A setting changed in one of these plugins, sometimes accidentally, sometimes left over from an old configuration, can introduce a block that has nothing to do with the actual robots.txt file contents you might check manually.<\/p>\n\n\n\n
Both plugins also include separate per-post and per-page noindex controls, which produce a different, related but distinct issue from a robots.txt block, worth keeping in mind if the affected pages are more scattered than a clean, predictable pattern.<\/p>\n\n\n\n
Step by Step Fix<\/h2>\n\n\n\n
Step 1: Confirm the Issue in Google Search Console<\/h3>\n\n\n\n
Open Search Console, go to the Pages report under Indexing, and look for \u201cBlocked by robots.txt\u201d in the list of reasons pages are not indexed. Click into it to see the full list of affected URLs.<\/p>\n\n\n\n
Step 2: Check the WordPress Reading Settings First<\/h3>\n\n\n\n
Go to Settings, Reading in your WordPress dashboard and confirm that \u201cDiscourage search engines from indexing this site\u201d is unchecked. This single setting resolves a surprising share of these cases and takes seconds to check.<\/p>\n\n\n\n
Step 3: View Your Live Robots.txt File Directly<\/h3>\n\n\n\n
Visit yourdomain.com\/robots.txt in a browser to see exactly what Googlebot receives. Look for any \u201cDisallow: \/\u201d line, which blocks your entire site, or more specific Disallow rules blocking directories or pages you actually want indexed.<\/p>\n\n\n\n
Step 4: Use the URL Inspection Tool for a Specific Page<\/h3>\n\n\n\n
If only certain pages are affected rather than the whole site, paste the specific URL into Search Console’s URL Inspection tool. It will confirm the exact block and often show which rule in your robots.txt is responsible.<\/p>\n\n\n\n
Step 5: Edit the Robots.txt File<\/h3>\n\n\n\n
If you are using Yoast SEO or Rank Math, both include a built-in robots.txt editor under their tools settings, letting you make changes directly inside WordPress without needing FTP access. Remove or adjust the specific Disallow rule causing the problem, being careful not to remove rules that are intentionally blocking sensitive areas like admin pages.<\/p>\n\n\n\n
If your site does not use one of these plugins, or if a physical robots.txt file already exists on your server overriding WordPress’s virtual version, you will need to edit it directly via FTP or your hosting file manager. Connect to your site’s root directory, locate robots.txt, and open it in a plain text editor. A safe, standard configuration for most WordPress sites looks like this, allowing all crawlers while still protecting the admin area:<\/p>\n\n\n\n
User-agent: *<\/p>\n\n\n\n
Disallow: \/wp-admin\/<\/p>\n\n\n\n
Allow: \/wp-admin\/admin-ajax.php<\/p>\n\n\n\n
A completely open configuration, appropriate once you have confirmed nothing sensitive needs blocking, simply reads:<\/p>\n\n\n\n
User-agent: *<\/p>\n\n\n\n
Disallow:<\/p>\n\n\n\n
Avoid using \u201cDisallow: \/\u201d alone, since that single line blocks your entire site from every crawler that respects robots.txt, a mistake that is easy to make and easy to overlook once it is in place.<\/p>\n\n\n\n
Step 6: Request Indexing to Confirm the Fix<\/h3>\n\n\n\n
Once the block is removed, go back to the URL Inspection tool, enter the affected URL, and click Request Indexing. This prompts Googlebot to recrawl the page sooner rather than waiting for its next scheduled visit.<\/p>\n\n\n\n
Step 7: Monitor Search Console Over the Following Days<\/h3>\n\n\n\n
Check back over the next several days to confirm the page moves from \u201cBlocked by robots.txt\u201d to properly indexed. If you are dealing with several pages, our guide on why a WordPress page might be stuck on Discovered, not indexed<\/a> covers what to do if pages remain unindexed even after the block itself is resolved.<\/p>\n\n\n\n
Quick Diagnostic Checklist<\/h2>\n\n\n\n
Symptom<\/strong><\/td> Likely Cause<\/strong><\/td> Where to Check<\/strong><\/td><\/tr> Entire site blocked, launched recently<\/td> Discourage search engines setting<\/td> Settings, Reading<\/td><\/tr> Site worked fine, broke after a migration<\/td> Staging robots.txt copied to production<\/td> yourdomain.com\/robots.txt<\/td><\/tr> Started after installing a security plugin<\/td> Plugin-added Disallow rule<\/td> Security plugin settings<\/td><\/tr> Robots.txt looks correct but site still blocked<\/td> Hosting provider virtual robots.txt<\/td> Contact hosting support<\/td><\/tr> Only specific pages blocked, not the whole site<\/td> SEO plugin per-page or per-section rule<\/td> Yoast or Rank Math settings<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n How to Prevent This From Happening Again<\/h2>\n\n\n\n
- \n
- Add a launch day checklist that specifically includes unchecking the Discourage search engines setting before going live.<\/li>\n\n\n\n
- Review your robots.txt file directly, not just through a plugin’s settings screen, immediately after any site migration or major deployment.<\/li>\n\n\n\n
- Check robots.txt after installing or updating any security plugin, especially if it advertises bot-blocking or firewall features.<\/li>\n<\/ul>\n\n\n\n
Set a recurring reminder, monthly or quarterly, to view your live robots.txt file and confirm it still matches your intentions. Our roundup of free SEO tools<\/a> covers several options for keeping an eye on this alongside your broader technical SEO health.<\/p>\n\n\n\n
Frequently Asked Questions<\/h2>\n\n\n
- Step 2: Check the WordPress Reading Settings First<\/a>
- A Staging Site's Robots.txt Copied to Production<\/a>
- What Blocked by robots.txt Actually Means<\/a>