1. Computing

Do Search Engines Search The Entire Web?

The Web is incredibly huge; can a search engine see all that content?

By

Do Search Engines Search The Entire Web?

The Web is an incredibly huge, complicated, and ever-expanding entity. Is it possible for one tool - one search engine - to index, curate, and retrieve content from the Web in a  meaningful way? 

In a word, no. Search engines have millions of pages in their databases (Google alone indexes close to eight billion at the time of this writing), but none of them even come close to indexing the whole Web, let alone the Internet. What are some pages that they exclude?

  • The invisible web,, a vast repository of information that search engines don't have access to, such as databases
  • Private networks, called intranets, that are not actually hooked up to the Web
  • Forms, like ColdFusion or CGI
  • Password-protected sites, like a university library
  • Sites that intentionally, for various reasons, keep their information from being indexed by search engine spiders

Will the Web ever be searchable from just one platform? Judging from the exponential growth of the Web day after day, week after week, and year after year, the odds are against it. This is one reason why expert searchers tend to not rely on just one search engine for their Web search needs; one search engine can't deliver the fullest Web search experience that many people don't even realize that they are missing out on. It's smart to diversify your Web search streams; here are a few resources that can help you address that:

  • The Ultimate Search Engine List: A list of search engines from general to incredibly niche that you can use as a jumping off point for your Web-based exploration.
  • The Top Ten Search Engines: Ten of the most popular search engines around the world that deliver a consistent quality search experience.
  • How to Pick the Right Search Engine for Your Needs: What if you're not exactly sure what it is you need to find what you're looking for? This article provides a walk-through of the steps you can take to determine that.
  • Search Engine Quiz: How much do you really know about search engines? Probably more than you think! Take this quick quiz to find out.
  • The Top Ten Most Useful Google Services: Google is consistently the most used search engine in the world for a reason - but it does a lot more than just search the Web. Find out more about Google's peripheral services, from Gmail to Google Scholar. 

©2014 About.com. All rights reserved.