The Invisible Web - Is My Site Getting Indexed?

September 14, 2005 – 3:00 am

Did you know that the great majority of web pages are not indexed and visible on the web? It is estimated that there are an enormous amount of pages that are never indexed. One estimate states that only 0.03 percent of all web pages have been indexed by search engines!

Is My Site Getting Indexed?

Getting your site indexed by the search engines is important. But it can be challenging! They key is to make sure your whole site is designed with the search engine spiders in mind. So if you are planning a new site make sure your web designer is clued up on designing for users and also for search engines. Unfortunately it is the case that many websites have been designed for the end user but not for search engines. The unfortunate result is that there are often indexing problems on sites.

How do you check whether your site has been indexed? Within Google use the site: operator as a search query. The query site:siteURL shows you how many pages have been indexed. For example, site:http://www.yoursite.com will show you how many pages of your site have been indexed.

What does it mean if your site has not been indexed?

It could be a brand new site that has never been indexed by the search engines. It could also be site that has been banned by the search engines because of spamming or rule violation. Or there could an existing problem that is actually stopping the search engines from indexing your site.

If you have a brand new site the key is to get links to your site from other sites. Forget about these offers of ‘We will submit your site to xxxxx thousand search engines for $xxx.’ These offers can often do more harm than good and it is speculated that search engines may penalize your site for using them. So start building links to your site from other reputable sites. Quick ways to get links include through optimized press releases, published articles, directories, and also blog entries.

If you are banned it means either you have been persistently violating search engine rules or you have employed someone who is involved in this. It is your site and your responsibility to know what your staff or consultants are doing with your site. Make sure they are not involved in using spamming techniques as it requires a lot of effort to get re-instated. It can of course also adversely affect your business.

Can The Spiders Index Your Site?

To find out whether the search engine spiders are able to index your site take a look at your log files and look for visits by Googlebot, MSNbot and Slurp the Yahoo spider. If there are no visits from the spiders it could suggest that there are no links to your site (or the links you do have are bad) and the search engines are unaware of the existence of your site. It could also mean there is some type of spider trap in existence that is preventing spiders from crawling your site.

A spider trap usually takes the form of some sort of technical issue. This may mean that your site displays well enough in your browser but there are some issues within the coding or design of the site. These problems can be caused by pop-up windows, flash or frame sites, JavaScript navigation, ineffective redirects, visitor passwords, wrong robot controls, or problems with dynamic URLs. You may want to talk with your designer regarding these issues or consult with an SEO expert.

Sitemaps!

The key to making your site accessible to spiders involves building site maps linking to all your site pages and also ensuring that text links are in existence to aid navigation between pages. Site maps are basically pages that include descriptive text links to all the other pages on your site. They make it easy for the spider to find all the pages on your site. If your site map has over 100 links it is advisable to have a multi-page site map – make sure they are all interlink. Remember also to optimize these pages too, as the descriptive links include lots of relevant information about the nature of your business.

Bottom Line

Unless search engine spiders can find your site there is no way for human visitors to find it using search engines either! Indexing is always the key first step in organic search engine optimization. Make sure this in place and you are ready to move forward.

Do not add to the invisible web statistics!