Let Search Engines Index The Site
The Discourage search engines from indexing this site setting tells every crawler to stay away, and it's most often found switched on by accident after launch.
How common is this?
- 3.3% of the WordPress sites we have this data for fail this check. Platform: WordPress.
Measured across the sites we audit, on each site's most recent snapshot.
What this check and mySites.guru tool looks at on your site
This check reads the “Search engine visibility” setting under Settings → Reading in wp-admin and fails if “Discourage search engines from indexing this site” is switched on.
The traffic you never knew you lost
Most of the checks in this reference are about stopping an attacker doing something. This one isn’t. Get this setting wrong and nothing gets hacked, nothing is exposed, no data leaks. What happens instead is quieter and, for a business relying on the site for traffic, arguably just as costly: the site simply stops appearing in search results, and nobody notices until they go looking for why enquiries have dried up.
The setting exists for a good reason. While a site is being built, or while a staging copy sits at a public URL for a client to review, you don’t want Google indexing half-finished pages, placeholder text, or a duplicate of the live site competing with it for rankings. WordPress makes it a single checkbox precisely so it’s easy to switch on during that phase. The trouble is symmetrical: it’s just as easy to forget to switch back off. A site launches, the checkbox stays ticked from the build phase, and every page on the site starts telling Google, Bing and every other crawler not to bother indexing it. Traffic that should be building steadily from launch day never arrives, and because nothing is broken in any way a visitor would notice, it can go unnoticed for weeks.
This is exactly the kind of failure that’s invisible from inside the CMS admin and only obvious from outside it, which is why an automated check catches it faster than most people would spot it themselves. It also tends to survive a migration. Plugins and manual processes that clone a staging site into production often copy the entire wp_options table along with everything else, checkbox included, so a site can go live already carrying the setting from a build environment nobody thought to check.
It’s also worth being clear about what this setting is not. It has nothing to do with security or privacy. A noindex instruction is a polite request that well-behaved crawlers, Google, Bing, and the rest, choose to honour. It does nothing whatsoever to stop a determined scraper, a bot with no interest in the standard, or a person who simply has the URL and types it into their browser. Anyone treating this checkbox as a way to keep a page hidden or private is relying on something it was never built to provide.
Unchecked, except on sites meant to stay hidden
For any site meant to be found, “Discourage search engines from indexing this site” should be unchecked. The only sites where leaving it checked is correct are ones deliberately kept out of search, a staging environment, an internal tool, a site genuinely not ready to launch yet.
WordPress implements this by adding <meta name="robots" content="noindex,nofollow"/> to the <head> of every page and by changing what its own virtual robots.txt output disallows. Some hosts and security plugins go further and add an X-Robots-Tag: noindex HTTP header as well, which works the same way but can’t be missed by a crawler that, for whatever reason, doesn’t parse the page’s HTML. Either mechanism achieves the same outcome, so a correct site should show neither.
How to fix it
- Log into
https://yoursite.com/wp-adminand go to Settings → Reading. - Find “Search engine visibility” and make sure “Discourage search engines from indexing this site” is unchecked.
- Click Save Changes.
- Confirm it’s working: view your site’s source and check that the
<head>no longer contains<meta name="robots" content="noindex,nofollow"/>. - Check the actual
robots.txtfile in your site root too. WordPress’s own reading setting only controls its virtual, generatedrobots.txt, and if a real, physicalrobots.txtfile exists in the site root, it takes priority and can keep blocking crawlers even after this checkbox is fixed. - If the site has been out of the index for a while, submit it for recrawling through Google Search Console rather than just waiting.
If you build sites for clients regularly, the safest habit is to make unchecking this the very last item on your launch checklist, done and verified after DNS has switched, not before. A setting flipped the moment a site goes live is a setting that’s easy to forget entirely once the project moves on to the next thing.
What mySites.guru does about it
mySites.guru checks every connected WordPress site’s search engine visibility setting on each snapshot, twice a day, and can switch it back to indexable with one click, across every connected site at once, rather than checking each dashboard by hand. That matters most in exactly the scenario above: an agency managing dozens of client launches has no realistic way to remember to check this box on every single one, and a missed one can sit undiscovered for months.
Let Search Engines Index The Site
mySites.guru checks every connected site for this automatically and flags it the moment it appears. These run twice a day on every connected site.
It can also fix this across every connected site with one click.