How to Stop Search Engines from Crawling a WordPress Site

Share on facebook
Share on twitter
Share on email

This guide teaches how to stop search engines like Google from crawling a WordPress site. Yes, you have control over what content these tech giants can and cannot access and index.

Why Prevent Search Engines Crawling WordPress?

Most WordPress site owners dream of making their site a huge hit with visitors. And the more free, organic search traffic you can get, the less you have to pay to promote the project.

But that can only happen if the search engines can crawl and index your posts and pages. So why would anyone want to block search engine spiders from crawling their web content?

There are 3 reasons why you might want to stop search engines crawling your site.

  1. Private or member-only blogs and websites
  2. Sites under construction
  3. Intranet (a private network)

#1 Private or member-only blogs and websites

Lots of people run private sites today. They offer a safe environment that only family, friends, or a group have access to. These projects are typically password protected and blocked from public view. The only people who need to know about a private site are those invited by the owner. Keeping them out of the search engine results pages (SERPs) maintains privacy. 

You can read more about making WordPress sites private here.

#2 Sites under construction

There can be plenty of trial, error, and mind-changing when building a new WordPress website or blog. You don’t want the test pages of a site under construction crawled and indexed by the search engines. It’s bad for the visitor experience and may even hurt your SEO. It’s better to keep sites under construction out of the SERPs.

You can do this with a staging site or local development environment.

A staging site is an exact copy or clone of your website that you work on in the background. When you’re finished working on the clone, you upload the changes to the live site. Most WordPress web hosting accounts offer 1-click staging environments for customers.

A local development environment is where you construct your website OFFLINE from a local computer. You upload the files to your server once you’re happy with the new build.

Your website is not accessible by search engine spiders in either of these cases.

#3 Intranet and project management

An intranet is a closed online network of connected computers to share information between users. Organisations and businesses typically use these private networks for internal communications. You don’t want to see an intranet in the SERPs. Project management is similar to an intranet but connects a tight-knit team rather than an office community.

How to Block Search Engine Spiders

You could try not linking to your site or specific pages you don’t want in the SERPs. But that only works if no one else shares those links. So, the only guaranteed way to keep your site out of the SERPs is to block access. There are three approaches to this.

  1. Block search engine visibility from WP admin
  2. Password protect the entire site via your cPanel console
  3. Password protect all or parts of the site via a plugin

Protecting sites and blogs with passwords is the best approach. It locks people and search engine bots out of your project. But let’s look at the WordPress built-in feature first.

#1 Discourage search engines from crawling your site

The most straightforward approach is also the weakest. Even so, major search engines should keep their web crawlers away, but it’s an optional response. That means they can choose to acknowledge or ignore the request, so there are no guarantees.

Log in to your WP admin area.

Go to Settings => Reading from your Dashboard side menu.

Scroll downwards to the Search Engine Visibility section.

Put a tick (check) in the box next to Discourage search engines from indexing this site.

Note: WordPress reminds you that it’s up to search engines to honour this request.

Click Save Changes to initiate the request.

How it works

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut elit tellus, luctus nec ullamcorper mattis, pulvinar dapibus leo.

Add Your Heading Text Here

A couple of things happen when you save changes. One is that WordPress adds the following line of code to your website or blog’s header:

<meta name=’robots’ content=’noindex,follow’ />

It also modifies your robots.txt file with this piece of code:

User-agent: *
Disallow: /

Search engine web crawlers or robots (bots) should now stop indexing your WP posts and pages. However, you might still come across random images or pages in the search results. You may not too, but the potential is there. That’s why password protection is a preferred option for blocking unwanted access.

#2 Password protect the entire site via cPanel

Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut elit tellus, luctus nec ullamcorper mattis, pulvinar dapibus leo.

Your WordPress web hosting provider should have provided you with cPanel access. The details for this are typically sent in the Welcome email. This tutorial assumes you’re familiar with cPanel.

If not, you might like to read this quick beginner guide to cPanel before continuing.

There are lots of tools for managing your WP hosting account in cPanel (control panel). That includes an option to password protect specific directories (folders) or the entire website.

Log in to your WP cPanel account.

Click Directory Privacy from inside the FILES panel.

Tip: If you don’t see it in FILES, check Password Protect Directories from the SECURITY panel.

You’re presented with a list of your site’s directories (folders) from the Directory Privacy screen.

Navigate to the root directory to access your WordPress files. It’s typically the public_html.

Click the Edit button from the right-side Actions column.

The next screen lets you set a password for your site’s root directory. That includes all files, folders, and subfolders. In other words, your entire site.

  1. Put a tick (check) into the Password protect this directory box
  2. Enter a name for the protected directory or leave the default
  3. Click the Save button

cPanel displays a Success message.

Now click the Go Back link.

The Security Settings screen now has a Create User section for you to create login details.

Enter a Username and strong Password.

Tip: Use the password generator if you need help to create a strong password. Make sure you store your new login details somewhere safe.

Click Save when you’re done.

cPanel displays a password set success message.

Tip: You can change or delete the authorised user login at any time from the Directory Privacy screen. Additionally, you can create separate logins for other users.

Click the Go Back link.

Your public_html directory now shows a padlock, and the Private status is set to Yes.

Your web browser now presents a login box when trying to visit any part of your website.

Unauthorised users who click Cancel will get a 401 Unauthorized error.

You can make the folder (site) public again at any time from the Directory Privacy screen.

Click the Edit link next to the site’s public_html folder.

Remove the tick from the Password protect this directory box, then click Save.

This action also removes the user account set up previously.

cPanel displays a ’removed access permissions’ success message.

Click the Go Back link.

Click the Back link on the next screen to return to the Directory Privacy list.

The padlock symbol has gone from the public_html folder, and the Private Status is reset to No.

Your site is now publicly accessible again.

#3 Password protect your site using a plugin

A few plugins can password protect a WordPress website or blog with a couple of mouse clicks. The one used to illustrate is the Password Protected plugin by Ben Huson. It hides your entire website behind a simple login box like the one below.

Read the beginner’s plugin tutorial first if you’re new to WordPress.

Log in to your WP admin area.

Install and activate the Password Protected plugin by Ben Huson.

Click the activated plugin’s Settings link from the plugins screen.

You can also access the Settings screen at any time from your Dashboard side menu.

Go to Settings => Password Protected.

You’re now at the Password Protected Settings screen.

A word about caching

Password Protected may not always work out of the box with WP sites that use caching. It’s possible to configure most setups by disabling caching for the plugin’s cookie.

Our test site uses a caching feature, so our plugin wouldn’t work at first. But Password Protected identified site caching and offered a workaround.

Here’s what the plugin picked up and suggested with the test site.

That’s a live link at the end that takes you to the W3 Total Cache.

Copy & paste the cookie name into the Rejected cookies box in the Advanced section.

The cookie needed for this is available on the settings screen:

bid_1_password_protected_auth

Click Save all settings at the end of the Advanced section.

You may use a different caching tool to W3, but the fix will be similar. Make sure you follow the plugin’s suggestions for any caching issues before continuing.

Now let’s explore the protected options on the Settings screen.

The first thing to do is put a tick (check) in the Password Protected Status box.

Selecting any of the optional options in the Protected Permissions section lets you whitelist those users or requests (see next image). That means users with those permissions can view your site as normal without needing to log in.

Next, type your password into the two New Password fields.

Tip: You can also change your password at any time from this screen.

The plugin also lets you whitelist IP addresses from the Settings screen if you want to. It even tells you what your current IP is. There are no limits on how many IPs you can add, but each one must go onto a separate line. Visitors to your site from a whitelisted IP don’t need to log in.

There’s a Remember Me section that adds a Remember Me checkbox to the login prompt. You can also determine the number of days you want the remember me feature to last.

Click Save Changes when you’re done.

Here’s what visitors to the test site will see if they try to open any posts or pages.

You’re done!

If you want to make the site public again at any time, remove the tick (check) from the Password Protected Status box and Save Changes.

That concludes this guide on the 3 easy ways to stop search engine bots from crawling and indexing your WordPress website.

Share on facebook
Share on twitter
Share on email

Subscribe to my newsletter

Subscribers to my newsletter get all the latest news, reviews, advice and gossip in the world of search engines, internet marketing, self-publishing, and North Wales (where I live). You can sign up for my free newsletter by clicking this link.

My Books

I have a range of webmaster related books for sale on Amazon in both Kindle and Paperback formats. WordPress, SEO & Self-Publishing.
On Amazon

My Online Courses

My courses cover a range of webmaster related skills, from learning WordPress itself, to installing an SSL certificate on your site.
On Udemy

Comments

Leave a Comment

Your email address will not be published. Required fields are marked *

Hello, I'm
Andy Williams!

You can get up to 90% off my Online Courses for webmasters, marketers & affiliates (plus a free course on Gutenberg).

Get my newsletter
delivered to your inbox

18748

We respect your privacy

18749