How to Noindex a Page on Your Website
What does it mean to noindex a page?
It’s essential to ensure the right pages on your site are crawlable, such as your homepage and key landing pages. Almost all the time, you want to ensure the hard work you’ve put into your web page is crawlable by search engines and viewable to people doing a search.
However, SEO best practice sometimes means it’s best to hide a page – such as internal search results, cart pages, or promotion terms and conditions pages – from the SERPs.
You can use a meta tag to prevent particular web pages from appearing in Google Search Engine Results Pages (SERPs) and prevent the indexing of certain types of content.
What are Robots Meta Directives and why are they important?
Robots meta directives are also known as “meta tags”, “meta robots”, or a “robots tag”. These pieces of code tell search engine crawlers how to crawl or index content.
When crawling a site, search engine bots such as GoogleBot will visit your website, take a look around and follow any linked pages. The bots then take the information they find and display it on search results.
Parameters that affect indexation: Noindex and Nofollow
Two ‘rules’ you can apply to pages include the “noindex tag” and “nofollow tag” directives. Usually, you can rely on Google to detect the essential pages to index, noindex, follow or nofollow. However, you can help suggest to Google URLs you would prefer not to be indexed, leaving more ‘crawl budget’ for your most important pages.
The robots meta tag looks like this in your page source code:
<meta name=”robots” content=”noindex,follow” />
Google makes the rules
Robots.txt file directives give Google bots suggestions on crawling and indexing a URL. But, they’re under no obligation to follow your meta directives. Even if you choose to use a robots.txt file, you may find any content you don’t want to display still appearing in search results for some time.
If you don’t want search engines to display specific content, you will need to use the noindex tag and allow search engines to crawl this content directive from the robots.txt file.
What is the difference between a meta robots tag and an x-robots-tag?
There are two main types of meta directives – meta robots and x-robots-tags.
Robots meta tags are crucial for SEO. By default, web crawlers will index your site and follow any links. The meta robots tag is an HTML code that allows people to control what search engines index and display in search results.
The x-robots-tag is an alternative way to manage how search engines crawl and index non-HTML files such as PDF files and images. X-robots-tags specify instructions in the HTTP header and enable you to control the indexing of a whole page, as well as specific elements.
Step-by-step instructions – How to Noindex a page
Two ways to noindex a page include copying and pasting the directive into the source code or using an SEO plugin such as Yoast:
Meta Robots Tag
<meta name=“robots“ content=“noindex“>
- Open the source code for the page you want to noindex.
- Copy and paste the above tag into a new line within the <head> section of your page’s HTML, or return a noindex header in the HTTP request.
- Save the updates, and your page will no longer appear in any search results.
To address Googlebot, Google’s standard web crawler (also known as a user agent), update your directive with the below example:
<meta name=”googlebot” content=”noindex” />
X-Robots-Tag
- For all .PDF files or images on your site, add the following tag to the HTTP response.
- To do this, you’ll need to open either your website’s header .php, .htaccess, or server access file. Here is an example for noindexing .PDF files:
<Files ~ “\.pdf$”>
Header set X-Robots-Tag “noindex”
</Files>
For XML-based Sitemaps, apply a “noindex x-robots-tag”. If you have only one XML Sitemap file, add the below to your .htaccess file:
<IfModule mod_rewrite.c>
<Files sitemap.xml>
Header set X-Robots-Tag “noindex”
</Files>
</IfModule>
Using Yoast SEO in WordPress
- Head to the “Advanced” section underneath the editing block of your posts.
- To apply the directive site-wide, go to the Yoast menu and to “Search Appearance”.
- Select “no” under “Show Categories in Search Results” to place a noindex tag on all category pages.
If you don’t have Yoast for WordPress, don’t worry – there are many other WordPress SEO plugins and resources with comparable functionality.
Robots Meta Directives – Best Practices and FAQs
When should I Noindex a page?
One reason to create this directive is to prevent web crawlers from duplicate indexing – for example if there is more than one version of a page. You may also want to noindex the following types of content:
- Posts or pages with little or no value for users
- Any page still in the staging environment
- A PPC landing page
- A Thank-you page
- Internal search results
- Promotions page, contests or product launches
What’s the difference between Noindex and Nofollow?
- Noindex tells search engine bots to crawl a web page but not add it to its search index to prevent it from appearing in SERPs. Use this meta tag on its own when you want the web crawlers to follow links, giving ranking authority to linked pages.
- Use a Nofollow tag to allow Google and other search engines to index a page in a search, but prevent bots from crawling any links.
- You can use a noindex directive alone or combined with a nofollow directive.
Do Noindex pages pass link juice?
“Link juice” is an SEO term used to refer to the value a site delivers to another through hyperlinks. Search engines such as Google see this as an endorsement from other websites that your content is worth promoting.
If you use the noindex directive and block a URL from search engines, it can still accumulate link juice. However, as the links won’t be visible, there is no way to “pass it on”. Have you disallowed a page that has already acquired link juice? Consider using noindex, follow tags to pass link juice along to other places on your website that may need it.
Ready to optimise your landing pages? Get in touch with Search Republic today and ask about our services – we’re here to help. Check out the rest of our blog for more SEO tips and resources!
