Search Engines play a vital role in website promotion. They are the first stop for most people trying to find information, services and products online. Therefore, it is necessary that your website surfaces quickly in search results.

There are number of search engines on the internet, some of which offer a service called “Paid Inclusion”.
Paid URL Inclusion is a search engine marketing technique where you need to pay a specific amount to search engine as an annual fee for your web page to be included in their index.

The question now is, why pay, when most search engines have automated programs called ’spiders’ or ‘crawlers’ that index all the pages on the web for free?
It is true that search engines will find your page and index them with no cost but WHEN? It could take at least couple of months for search engines to finally index the entire website. This is where paid URL inclusion beats Free Inclusion – Speed. It offers faster indexing of your web pages.

Search engines that provide paid URL inclusion services, use an additional spider that goes out and indexes specific pages that have been paid for. Free spiders will find a site, follow the links and index the pages, but it will take some time and you could lose out on valuable traffic but a paid spider will index the pages that have been paid for, almost immediately.

Advantages of Paid URL Inclusion:
Quick Inclusion – after the payment is made, the website is indexed by the spider instantaneously.
Rapid re-indexing – the spider will return to the website very often, sometimes even daily!

Disadvantages of Paid URL Inclusion:
Cost – The biggest disadvantage is the charge. The annual cost for a ten page website ranges from $170 for Fast/Lycos to $600 for AltaVista.
Reach – Another disadvantage is the limited reach of paid URL inclusions. Popular search engines like Google, Yahoo and AOL do not offer paid inclusion. This implies that Paid URL inclusion will amount to only a small fraction of the daily traffic to the website.

In the end, the fact remains that search engines will still index websites whether they are paid inclusions or not. But if you want immediate indexing for your website on AltaVista, Lycos etc then you could opt for paid URL inclusion.

Some webmasters recently experienced a delisting from the search engines without any good reason. The webmasters had not done anything wrong and their web sites were optimized for search engines. Nevertheless, the web sites had been removed from search engines.

Poorly configured firewalls can block search engine spiders

It turned out that the delisted web sites were all hosted by the same hosting company. More precisely, the web sites were all hosted by a hosting company that used a special firewall software by SonicWALL Inc.

That firewall stopped the search engine spiders from accessing their web sites. Google, Yahoo, MSN and all other search engines that request the robots.txt file couldn’t index the web site anymore because the firewall didn’t allow that:

“An attacker could retrieve robots.txt from the server, then use the contents of this file to discover the path of an unprotected administration interface for the server. The attacker may gain control of the webserver using this interface.

The information gathered from robots.txt could be used for system compromise and control of the web server.” (source)

This is the standard security settings of the SonicWALL firewall and it basically means that your web site won’t be spidered by search engines if you use this firewall without customizing it.

A firewall with these settings will drop the connection to anyone requesting the robots.txt file so that it looks as if the web site is offline. From an SEO point of view, this is very bad for your web site because all good search engine spiders request the robots.txt file before indexing your web site.

What does this mean to you?

If your web site is not listed on search engines although it has many good incoming links and optimized web page content, you should ask your web host if their firewall blocks search engines that request the robots.txt file. Your web host might not be aware of the problem.

Many webmasters don’t get high rankings on Google and other search engines just because the indexing robot has difficulty to index their web pages.Search engine robots are very simple software programs. If an indexing robot cannot find the content of your website immediately, it will skip your site and go to the next link in the list. For that reason, it is very important to make sure that search engine robots can index your web pages without problems.

Here are the top 5 elements that drive search engine robots away:

Element 1: Your robots.txt file is damaged or it contains a typo

If search engine robots misinterpret your robots.txt file, they might completely ignore your web pages.

Double check your robots.txt file and make sure that you use the disallow parameter only for web pages that you really don’t want to have indexed.

Element 2: Your URLs contain too many variables

URLs with many variables can cause problems with search engine robots. If your URLs contain too many variables, search engine robots might ignore your pages.

Here’s Google’s official statement about web pages with many variables:

“Google indexes dynamically generated webpages, including .asp pages, .php pages, and pages with question marks in their URLs. However, these pages can cause problems for our crawler and may be ignored.”

Element 3: You use session IDs in your URLs

Many search engines don’t index URLs that contain session IDs because they can lead to duplicate content problems. If possible, avoid session IDs in your URLs. Better use cookies to store session IDs.

Element 4: Your web pages contain too much code

Of course, your web pages can contain JavaScript code, CSS code and other script code that is not directly related to your content. Visit your website with a web browser and select “View source” or “View HTML source”.                                                                                                                                                               If it is difficult for you to spot the actual content of your website then search engines might also have difficulty to parse your pages.

Element 5: Your website navigation causes problems

Fancy JavaScript or DHTML menus cannot be parsed by most search engine robots. Flash or AJAX menus are even worse when it comes to website navigation.
As mentioned above, search engine robots are very simple programs. They can follow HTML links; all other links can cause problems.

Optimized web page content and good inbound links are crucial for high search engine rankings. However, the best content and the best links won’t help you much if search engines cannot index your pages.