First, what is a search engine?
Search engines are very complicated software programs. They use software programs called spiders to "crawl" all over the Internet, amassing links to content, multimedia, etc. that they then include in their search engine indexes. When you type a query into a search engine, the search engine is not searching the entire Web, it is searching its vast internal index.
Are there illegal search engines?
Search engines merely find content; they do not host the content. Therefore, there are no illegal search engines, per se. However, search engines certainly do come across content that is under copyright, has been illegally obtained, or is posted against criminal or civil law.
The term "illegal search engines" is most heard in tandem with "torrent search engines", which, while a great source for finding and downloading legal, public domain content, also are a prime location to find all sorts of copyrighted material, which, technically, under the laws of most countries, are illegal to download without proper payment and through pre-approved channels.