What if you find out a major section of your website is not visible to Google at all! Worse can be that the complete site not indexed. Website indexation is a major concern for the webmasters across the globe.
There are two scenarios where the above mentioned problem may arise. They are as follows –
1. New Site:
- Your new website is not indexed by Google even after the website is pushed live.
- Some of the possible reasons are mentioned below:
2. Old site:
- Your website was already indexed by Google however suddenly it has disappeared from Google’s index altogether.
- Tip: To check whether your site has been indexed by Google or not, do site: your domain name. The result should match the number of pages on your website.
- You have created new content pages for your website but they are not indexed by Google.
- Below are some of the factors that may cause the problem:
Some reasons as to why your website is not indexed by Google or has been removed from indexis explained in detail.
- Commands like Disallow instruct search engines not to crawl specified URL’s or Sub-directories within a website.
- In case of a new site, you would have disallowed all search engines from crawling your website during the development phase of the website.
- Just change the command from Disallow to Allow once your site goes live and ping your website URL to Google or submit a XML Sitemap of your website through webmaster tools for inclusion in Google’s index.
- The following example illustrates Robots.txt file for the above case:
In development phase:
After Site goes Live:
- In case of old site check whether the section is blocked via Robots.txt or not. Follow the above mentioned steps if it was blocked previously.
- Different types of crawl errors are as follows:
- DNS: Google is not able to communicate with the DNS Server which may be caused if your DNS server is down or there is an issue with the DNS routing to your domain.
- Server: Either the server is taking too much time to respond or it is blocking Google.
- URL: Problem occurred when Google tried to crawl the respective URL due to broken links or page did not exist.
- Syntax Errors in Robots.txt file: Robots.txt file errors are caused when the file exists on the website but it is unreachable due to improper use of syntax.
- Once the errors are fixed you can:
- Ping Google using fetch as Google option
- Mark them as fixed once the proper 301 redirect is in place
- Re-submit the updated Sitemap
Meta Robots Tag:
- You can tell Googlebot not to crawl that page or any links present on that page using metarobots tag.
- During the development phase of the website, we add content attribute as “noindex, nofollow” in order to prevent search engine bots from crawling the webpage. It is illustrated as follows:
- Remove the above tag from all the pages of website once the site goes live and submit your website URL to Google or submit a Sitemap of your website through webmaster tools for inclusion in Google’s index.
- In case of old sites also, follow the above steps if the above tag is present on pages you want Google to index.
- Spider stopper is a term used for things that would restrict the search engines from crawling the webpage or website either partly or completely.
- You might have blocked your site using .htaccess if you are using Apache Server or through administrative console if you are using IIS Server.
- If you are using Content Management System (CMS) to manage your content within the website, then check whether your privacy settings is properly configured or not.
New domain migration:
- If you shifted your website to a new domain name there are many things that you would need to consider such as internal and external linking, redirects, etc.
- Proper redirects should be implemented, i.e. home page from old domain should be redirected to home page of new domain, product page from old domain should be redirected respective product page of new domain and so on.
- It is recommended that redirects are implemented in one to one fashion so that the SEO value of old domain is retained and passed on to the new domain.
- Changes should be made to external and internal linking structure in accordance with the new domain.
- Add the new domain name to your Google Webmaster Tools account and monitor both old and new domain. Check whether the implemented redirects work properly or if there are any broken links present. Check whether new site does not show any unwanted 404 errors.
- When your website contains lots of links to other domains that may be out of context and without “rel=nofollow” attribute, then Google might term this links as unnatural and might take manual actions against the particular page containing those links or the entire website.
- If your website is removed from index, check your Google Webmaster Tools account to see if manual actions has been taken against you. If so, you may receive the message illustrated below:
- If manual action has been taken remove all the unnatural links or add “rel=nofollow” attribute and request reconsideration of your site.
- The same should be done if you get message saying unnatural links pointing to your site.
- As a precaution all the outbound and inbound links should contain “rel=nofollow” attribute.
- If the domain you have purchased was used before and was marked up as shady domain by Google and manual action was taken against it, your website may not show up in Google’s index.
- Thus, in this case if your website is not indexed by Google,request reconsideration of your website.
- If the previous owner has excluded some URL’s using URL removal tool in Google Webmaster Tools and if you are using the same URL for your new website, it won’t be indexed by Google as well. For example:
- “example.com/about-us” is removed by previous owner than your about us page with same URL will not be included as well.
- For including the URL again submit your website URL to Google or submit a Sitemap of your website through webmaster tools for inclusion in Google’s index.
Issues with Sitemap:
- If there are issues in the Sitemap that you might have submitted through webmaster tools, there are probable chances that some of the pages might not get indexed.
- There might be issues in your syntax or you may have entered wrong/broken URL’s.You can check up your Webmaster Tools account to find out the issues arising in your Sitemap. The syntax error is illustrated as follows:
- Fix those issues and resubmit your Sitemap in Google Webmaster Tools.
Impact of Duplicate Content on our Website:
- Duplicate content refers to content within the website or across domains that is exactly same or is relatively similar.
- This problem may occur, if your website has multiple domain versions, i.e. www and non-www versions.
- Google may not index all the duplicate pages or will not index the website at all. It will show only those pages which it thinks is more appropriate.
- If your website contains duplicate content or multiple domain versions, use:
- rel=”canonical” link element to tell Google about the duplicate content pages and your preferred URL
- URL parameter handling tool
- 301 redirects to redirect to preferred domain version.
- Duplicate content issues on a website can be illustrated as follows:
- If your website has been removed completely, then make the above changes and submit your site for reconsideration.
- For more details regarding duplicate content guidelines, click here.
Site Loading Time:
- If your website takes too much time to load, it may be a reason why your website is not indexed by Google.
- Your webpage or website needs to be loaded successfully for Google to index the same.
- One of the reason why your website takes too much time to load may be that your hosting server is down.
- This issue can be resolved by contacting the hosting provider.
The above mentioned points are the major issues. However, there are many other issues that might have resulted in removal of your website from Google’s index. If your site is not indexed by Google, don’t worry, just check for above issues and make the recommended changes.