Webmasters frequently make mistakes with directive tags — HTML snippets that tell search engines how to crawl (and how not to crawl) a website. We draw attention to these issues when performing SEO audits and content audits, but the simple truth is that directives aren’t complicated, and we’d rather not spend our workdays worrying about them.
If you’re reading this blog, you’re in the same boat. We’ll skip past the basics and assume that you have some confusion about noindexing pages, and you’re concerned that you’ll make a mistake that costs you ranking position. Take a deep breath; search engines are fairly adept at figuring out webmaster intent, and even if you’ve made a serious mistake, fixing it is usually a fairly simple process.
Here’s how Google defines a noindex tag. We’ll warn you, it’s a pretty simple explanation:
noindex – Prevents the page from being indexed.
That was straightforward, right? The noindex tag sends a signal to the search engines telling them that you don’t want the page to appear in search. You might use this tag on category pages, duplicated pages, PPC landing pages, or other pages that you’d rather not see in search.
There’s a caveat, though, and it’s a big one. Google (and the other search engines) will sometimes ignore a noindex tag, indexing the page in question and showing it in search.
Why? We have no idea, but the folks at Google (and, again, the other search engines) leave a lot of room for webmaster error. Most website owners use tags improperly or inconsistently, so the search engines use directives as suggestions, not hard-and-fast rules.
We ran into this issue several years ago with a client’s store page, which another SEO firm had outfitted with noindex tags. The store was still showing up in search, but for specific reasons, the client didn’t want that — they didn’t want the store showing up until a customer actually clicked on the requisite link.
The solution was to block search engines from accessing the store via the htaccess file (if you’re dealing with something similar and you don’t immediately know what we mean, ask your web developer or drop us a line). We then used Google Webmaster Tools to request that the page be de-indexed.
The problem was resolved, but the client was lucky; Google alone decides whether or not to send traffic to a page. Once something is indexed, getting it scrubbed from the search rankings can be really difficult.
The entire purpose of a search directive is to direct search. With that in mind, why would Google ignore a noindex tag? We can think of a few reasons:
In all of these scenarios (well, except the last one), the search engine made the determination that the page was too important for the “noindex” tag to be legitimate; it must have been a mistake.
So, here’s the moral of the story: If you really, really don’t want a page to show up on search, the noindex tag isn’t enough. You’ll need to block the search spiders from accessing the page, then add the noindex tag, then make sure that your sitemap is up to date, then monitor regularly to make sure that the page doesn’t pop up.
If you don’t want a page to appear on search, but it’s not a life-or-death situation, go ahead and use noindex tags. As a general rule, though, webmasters should avoid using the noindex tag unless they have a good reason to use it. Let the search engine decide what should rank. They’ll figure it out.
Of course, most of our clients have the opposite problem — they want pages to appear on search, but they’re unable to raise their domain authority and capture that sweet, sweet organic traffic that they know is lurking out there. The solution to this problem is to perform keyword research and create an effective content strategy, ideally while addressing technical issues along the way. We can help with that, too.