Why Do We Need Search Engines?
For the same reason you need a card catalog in a library. There is lots of great and useful information in a library, but it’s physically impossible to examine all the books personally. Not even the most indefatigable web-surfer could hyperlink to all the documents in the aptly named World Wide Web. There are billions of pages on the Web. And every minute of the day, folks are posting more.
Q. If it’s impossible to examine all the documents on the Web, how do the search engines do it?
A. They use software programs known as robots, spiders or crawlers. A robot is a piece of software that automatically follows hyperlinks from one document to the next around the Web. When a robot discovers a new site, it sends information back to its main site to be indexed. Because Web documents are one of the least static forms of publishing (i.e., they change a lot), robots also update previously cataloged sites. How quickly and comprehensively they carry out these tasks varies from one search engine to the next.