A (Brief) History of Search Engines

Many people will not even know the name “Gerard Salton” but he is considered by many to be the father of modern search engines.

This is because, while he worked at Cornell University, he developed the SMART (System for the Mechanical Analysis and Retrieval of Text) Information Retrieval System.  It was also from this system that the term “Hypertext” was coined to refer to sets of related text where semantically similar pieces of text are automatically linked.

Project Xanadu, founded in 1960 by Ted Nelson, an American sociologist and philosopher, took Salton’s research to the next level.  Project Xanadu was created with the goal of creating a computer network with a simple user interface. While Nelson never completed his project, it was considered to be one of the basic building blocks of the internet as we know it today.  Ted Nelson went on to co-found the Itty bitty machine company, or, as you may know it today:  IBM.

The First Computer Network

Flash forward to the late 60s where researchers were working on a way to allow for uninterrupted communications between locations.  The department in charge of this project is called the Advanced Research Projects Agency – also called ARPA. ARPA developed ARPANET – the first packet switching network. 

Unlike a traditional telephone line where each line was used for a single call, packet switching could use a similar line to send out multiple packets to multiple locations as needed.  Charles Herzfeld, ARPA Director (1965–1967), said:

“The ARPANET was not started to create a Command and Control System that would survive a nuclear attack, as many now claim. To build such a system was, clearly, a major military need, but it was not ARPA’s mission to do this; in fact, we would have been severely criticized had we tried. Rather, the ARPANET came out of our frustration that there were only a limited number of large, powerful research computers in the country, and that many research investigators, who should have access to them, were geographically separated from them.”

In other words, the first computer network.

A Canuck Develops the First Search Engine

While it was the American research society that devised the techniques for allowing communication between 2 locations, it was a Canadian who developed the first search engine.  In 1990, a McGill University student by the name of Alan Emtage, created the first search engine, called Archie. The original intent of the name was “archives,” but it was shortened to Archie so that it could follow the Unix naming standards of the time.

Archie used different components to help people find information on public FTP servers.  It contained the equivalent of a search engine spider – a series of scripts designed to hunt down information;  It contained an Index, or central database of the information stored, and a way to enter queries into an interface to match documents stored in the index.

While Archie’s purpose was to find and index information found on FTP servers, 2 other search engines were also developed soon after:

Veronica (1992) was a search engine system for the Gopher protocol, released in November 1992 by Steven Foster and Fred Barrie at the University of Nevada, Reno.

Jughead (1993) is a search engine system for the Gopher protocol. It is distinct from Veronica in that it searches a single server at a time.  Jughead was developed by Rhett Jones in 1993 at the University of Utah.

Tying It All Together – The W3C Consortium

There were other university projects that came later that functioned in a similar fashion as Archie, Veronica and Jughead, but, as one would expect, these systems were largely independent of each other.

It wasn’t until 1994 when Tim Berners-Lee formed the W3C – the World Wide Web Consortium as a way to tie these disparate systems together and allow them to communicate with one another using common protocols.  Berners-Lee also created the Virtual Library, in 1991 (shortly after Archie) which is the oldest catalogue of the web.  While Berners-Lee is considered “the father of the Internet” he says he just had the idea to take hypertext “and connect it to the TCP and DNS ideas and — ta-da! — the World Wide Web

Things Move Quickly

While the idea of the W3C was being formulated, different search engines began to spring up.

 

In 1993, Matthew Gray developed The World Wide Web Wanderer, also referred to as just the Wanderer.  It was a Perl-based web crawler that was first deployed in June 1993 to measure the size of the World Wide Web.  The crawler was used to generate an index called the Wandex later in 1993. Really, Wanderer was the first search engine crawler.  

 

Also in 1993, six undergraduate students at Stanford University were looking for a way to use statistical analysis of word relationships to improve relevancy of searches on the Internet.  The search engine Excite was born out of this research and launched in 1995.  At its height, Excite was one of the most recognized brands online and by 2000 it was the fourth most visited website on the internet.  At its most popular, Excite provided an assortment of content including news and weather, a metasearch engine, web-based email, instant messaging, stock quotes, and a customizable homepage for its users.

Not really a search engine initially, In 1994 Stanford University students Jerry Wang and David Filo created Yahoo!  It was originally an Internet bookmark list and directory of interesting sites.  At its peak, Yahoo! provided its users a web portal which included: a search engine, business directory, web based email, news, finance, groups, answers, the first major online advertising platform, mapping, video sharing, fantasy sports, and its own social media website.  At one point Yahoo! was the most widely read news and media website – with over 7 billion views per month – ranking as the sixth-most-visited website globally in 2016.

In March 1994, the search engine called WebCrawler was born.  It was the first search engine to provide full text indexing and search capabilities.  By November 14, 1994, WebCrawler served its one millionth query. On June 1, 1995, a company called America Online (AOL) bought WebCrawler so that it too would have the capability to crawl and index the web.  

Also in 1994 Lycos was formed.  What was unique about Lycos was how quickly it could crawl and index the World Wide Web.  Lycos had the largest index at the end of 1996 with 60 million documents.

In 1995 a search engine named Altavista entered the market and it was even faster than the rest.  In August 1995, it conducted its first full-scale crawl of the web bringing back about 10 million pages.  Over 300,000 visitors used Altavista on its first day. They had 19 million hits by the end of 1996, and 80 million per day at the end of 1997. By 2000 the value of Altavista had reached $2.3 billion.  Altavista is also important because it became the first Internet search engine to offer image, audio, and video search in 2002.

In 1996, two Stanford University students test Backrub, a new search engine which ranks sites based on inbound link relevancy and popularity. Backrub would ultimately become Google in 1998.  The students names were Sergey Brin and Larry Page. 

In 1997 a search engine called “Ask Jeeves” came into existence.  It was unique because it allowed searchers to use questions to find websites and it was very good at doing this.  

Realizing that search wasn’t going away, Microsoft entered the fray in 1998 with its first attempt at a search engine called MSN search.  MSN Search was renamed Windows Live in 2006, then renamed again to Bing in 2009

AllTheWeb was launched in 1999 and it had even more features and speed than its predecessors.  It was later bought by Overture which was then purchased by Yahoo in 2003.

Throughout the 2000’s many of the smaller engines closed down or were bought by larger competitors.  

For example the following changes happened between 2000 and 2005

  • A search engine called LookSmart bought a competitor called Zeal
  • Looksmart also bought Wisenut
  • Excite went bankrupt and was sold to competitor Infospace.
  • Ask Jeeves bought smaller competitor Teoma, who just released their engine a year before.
  • Overture bought both Altavista and Alltheweb in 2003
  • Yahoo bought Inktomi, followed by Overture
  • Lycos was bought by the second largest Internet portal in Korea
  • And more.

The 2000’s were a tumultuous time for search.  Where there used to be 5 to 8 big players in search, things began to change.

Lately there has been only 2 or 3 major search engines in North America – Google, Yahoo and Bing.  One should note that Yahoo gets it search results from Bing. So essentially there are only 2 search engines one needs to concern themselves with – Google and Bing.

As one can see, we’ve gone through some changes in search.  Over time, in addition to consolidation in the industry, it has evolved to become a dominant force in our lives.

Now, in addition to “the big 3” we have other forms of search.  Mobile search has been growing in recent years, while voice search is just starting to “cross the chasm” to become used more often by searchers.  Over the next few years we expect to continue to see this sort of evolution of search.