Crawling and Indexing… What’s the Difference?

The terms crawling and indexing (and indexing’s cousin, caching) are frequently used together, but they are vastly different in terms of SEO.

Exact definitions probably differ from person to person, but following is how I explain the processes:

Crawling is the process of an engine requesting – and successfully downloading – a unique URL. Obstacles to crawling include no links to a URL, server downtime, robots exclusion, or using links (such as some JavaScript links) from which bots cannot find a valid URL.

Indexing is the result of successful crawling. I consider a URL to be indexed (by Google) when info: or cache: query produces a result, signifying the URL’s presence in the Google index. Obstacles to indexing can include duplication (the engine might decide to index only one version of content for which it finds many nearly identical URLs), unreliable server delivery (the engine may decide to not index a page that it can access during only one-third of its attempts), and so on.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>