Some webmasters recently experienced a delisting from the search engines without any good reason. The webmasters had not done anything wrong and their web sites were optimized for search engines. Nevertheless, the web sites had been removed from search engines.

Poorly configured firewalls can block search engine spiders

It turned out that the delisted web sites were all hosted by the same hosting company. More precisely, the web sites were all hosted by a hosting company that used a special firewall software by SonicWALL Inc.

That firewall stopped the search engine spiders from accessing their web sites. Google, Yahoo, MSN and all other search engines that request the robots.txt file couldn’t index the web site anymore because the firewall didn’t allow that:

“An attacker could retrieve robots.txt from the server, then use the contents of this file to discover the path of an unprotected administration interface for the server. The attacker may gain control of the webserver using this interface.

The information gathered from robots.txt could be used for system compromise and control of the web server.” (source)

This is the standard security settings of the SonicWALL firewall and it basically means that your web site won’t be spidered by search engines if you use this firewall without customizing it.

A firewall with these settings will drop the connection to anyone requesting the robots.txt file so that it looks as if the web site is offline. From an SEO point of view, this is very bad for your web site because all good search engine spiders request the robots.txt file before indexing your web site.

What does this mean to you?

If your web site is not listed on search engines although it has many good incoming links and optimized web page content, you should ask your web host if their firewall blocks search engines that request the robots.txt file. Your web host might not be aware of the problem.

I often get asked about web site submission to Google. Most people believe that submitting your site your Google ensures search engine rankings on arguably the best search engine today.

Well, the Google spider crawls through the internet and finds majority of sites via links from other sites. So optimizing your site and getting some great incoming links from directories and other related sites would ensure that your site is indexed by Google. If you still want to do a web site submission to Google, you can do that using their online form at http://www.google.com/addurl/.

One does not need to submit each individual page to Google. Submission of your web site’s front page will do that job. However, you need to make sure that your inner pages are linked well from the front page.

Another method to submit your web site to Google is through Google Sitemaps (now a part of Google’s webmaster tools). Web site submission to Google using sitemaps requires an XML file to be generated. This XML sitemap file contains all the URLs that you want crawled. It can also contain other data like the last modified date for each page, change frequency of each page and the priority of the page. Priority is the relative importance of that page as compared to the other pages of the site.

I’ll discuss the Google webmaster tools in details in another post. Hope this post has helped clear some myths of web site submissions to Google and helped understand the basics of indexing by Google.

Today, search engines play a very important role in getting traffic for a website. Even big sites like Amazon and ebay are trying to improve their search engine placement by working on their internal site structure and pages.

A case in point that shows how important search engine rankings are for bigger sites is ebay. WayBackmachine shows that the title of Ebay’s front page in Jan 2001 was “eBay - Your Personal Trading Community”. Compare this with today’s title that incorporates their primary keywords - “eBay - New & used electronics, cars, apparel, collectibles, sporting goods & more at low prices”.

Further, the ebay product pages are now all without querystrings (i.e. - all the pages have their URL rewritten thus enhancing spideribility).

There are a few very simple things that can be done to improve your search engine placement -

1 - Make sure you have unique titles on every page. The title should be crisp but should contain your keyword.

2 - Have a short meta description in place. Again this concise description should contain your keyword.

3 - Its best to use text links to link all your pages. If you have a flash or JavaScript navigation panel, try to link those pages using text links that can be placed at the bottom of your page.

4 - Write unique content. Spiders like to see unique content that offers information about your products and services.

5 - Upload pages close to the root directory. A simple site structure with pages close to the root (www.domain.com/directory/, www.domain.com/directory2/) enhances chances of getting spidered quickly.

6 - Get links from similar sites and link to sites that will offer value to your users

7 - Submit to DMoz and other major directories

Though just doing the above will not help you get search engine placements for really competitive keywords, they are steps essential for every site and would give you a good advantage over other sites right from the day your site is launched.