What Are Web Crawlers And Howe Are They Used
Have you ever wondered how a search engine knows all the websites in the world, or why it chose to rank one site over another? If this is something that you have been wondering about, then you will be happy to know that I am here with the answer. Search engines (like Google and Yahoo) use little bots called web crawlers. There job is pretty much described in their name. Pretty much, what they do is crawl around the internet and look for URL’s of new sites, and sites it has already been to. From here it will read all the links in that website, and keep it stored into its data base. Then these web pages, stored in the database, are used by search engines when you type in keywords. Now, this tells you what a web crawler is, but this still does not answer the question of how it ranks the sites.
Web crawlers do not just spend their time looking for new sites, they also spend their time going to sites they have already been to. Here they can see how many hits a website has received. The more popular a web page is, the more hits it gets, and the higher ranking it gets from a search engine. As you can probably guess, there are many web pages out there. How can one web crawler get to all of them? The answer is, they don’t. On one site, like Yahoo, they could have thousands of web crawlers looking all over the internet for sites and revisiting sites. However, as you can guess, a thousand web crawlers for millions of web pages still is not enough. This is why when you make a new site it’s hard to get it noticed by a search engine. There are ways that you can help yourself out. Most of these search engine sites have a box where you can input your URL to help the web crawlers find it. This, however, could still take weeks or months before they get to your site.
