Crawler-Based Search Engines

Crawler-based search engines, such as Google, MSN create their listings automatically. They "crawl" or "spider" the web, then people search through what they have found.
Crawler-based search engines have three major elements. First is the spider, also called the crawler. In case of google it is called googlebot. The spider visits a web page, reads it, and then follows links to other pages within the site. This is what it means when someone refers to a site being "spidered" or "crawled." The spider returns to the site on a regular basis, such as every month or two, to look for changes. If you change your web pages, crawler-based search engines eventually find these changes, and that can affect how you are listed. Page titles, body copy and other elements all play a role.
Everything the spider finds goes into the second part of the search engine, the index. The index, sometimes called the catalog, is like a giant book containing a copy of every web page that the spider finds. If a web page changes, then this book is updated with new information.
[search engine crawler]