Blocking The Search Engine Spiders From Indexing Specific Pages
Dylan Downhill
© 2005 Elixir Systems
The best way to ensure content is not indexed by the search engines is to not put it on the publicly accessible web in the first place. If you don't want personal details of your love life to be found then don't put them on the internet!
If you want to publish information for a select group (perhaps a new design) then if you don’t link to the new pages from anywhere then the spiders will not be able to find them.
You can put password protection on the pages - the search engines will not enter passwords and so not index the pages.
You can add a line to the robots.txt file blocking access to these pages (a good tutorial is here). This allows visitors to read the pages but will block spiders from reading them. This is also probably the best way to block whole directories. One down side is it makes it clear to anyone where your protected content is.
Advertisement
Four Corners Hotels - New Look, Same Low Prices!
For hotels from New York to Los Angeles, from Dublin to Dubai our prices can't be beat. Book online at www.fourcornershotels.com or through our 24 hour phone reservation center.
The robots meta tag is also available, just put the following line in the <head> section of each page:
<META NAME="ROBOTS" CONTENT="NOINDEX,NOFOLLOW">
The spiders will download the page, but upon finding the meta tag will ignore the page. This should be avoided where you don't want the spiders to download the page - due to resource usage issues for example.
Click for more information on the robots meta tag
The ‘rel="nofollow"’ tag is only supported by Google at present and so doesn't help protect content as well as the methods above. I mainly use it to block text links I don’t want to be found for (links saying ‘home’ for example).
For more information on search engine optimization contact Elixir Systems at 602.494.6326 or at http://www.elixirsystems.com.
You can comment on this article at the Elixir Systems SEO Blog
Learn More About
• Elixir Systems Online |
Newsletter Actions• Subscribe to Newsletter• Modify Subscription to Newsletter • Unsubscribe from Newsetter • Ask A Question |
This is the newsletter from Elixir Systems. Elixir Systems is a Phoenix, Arizona based search engine optimization and search engine marketing services company. If you received a copy of this newsletter and would like to subscribe go to http://www.elixirsystems.com/newsletter.
You are currently subscribed as %BASIC:EMAIL%
To unsubscribe go to http://www.elixirsystems.com/newsletter/unsubscribe.php
or email newsletter@elixirsystems.com with unsubscribe in the subject line.
© Elixir Systems, All rights reserved. You are f~ree to use this article in whole or in part, as long as you include complete attribution, including live web site link and email link.
Please also notify us where the material will appear using the address