Search Engine Technology:: The Three Kinds of Search

When someone mentions "search" most people immediately think of web search. There are actually three kinds of search which are attracting attention at the moment:

Desktop Search: The ability to search a PC for the information that is required, for a variety of different file types but across a single file system.

Intranet Search: The ability to search an Intranet (the network on the inside of a firewall) for a large number of file types across a number of different storage techniques.

Internet Search: The ability to search the entire World Wide Web for a limited set of file types across a limited number of storage techniques.

Microsoft and Google have offerings in all three platforms now. Microsoft has gone down the path developing Desktop Search, then Intranet Search (through the SharePoint Portal technologies) and recently launched Web Search. Google has done the reverse of Microsoft, by forging their business ahead on the back of their Web Search technology, followed by the Google Search Appliance (Intranet Search) and providing Desktop Search.

Contrast this with Yahoo! Yahoo! are a marketing company, who went out and purchased there Web Search product (buying Inktomi outright and picking up AltaVista and Fast through the purchase of pay per click company Overture). There desktop search product has been developed by the companies. However, Intranet Search remains off the radar for Yahoo! Yahoo! choose not to continue down this path by selling off the Enterprise Search division of AltaVista to Fast Enterprise Search.

To a user, the end result remains me the same - find me the information that I want quickly. However, to the search developer, the same query in all three is totally different to service and provides many technology and business issues that may not be first apparent.

Take for example searching for you name. On the web, this is going to return all websites that contain you or some other person named the same. Also great for the ego to see your own name up in lights for your respective 15 minutes of fame or reliving past glories.

Searching on your PC for your name will return all files that contain your name in the title and content. Pictures, Word documents, Excel spreadsheets, the presentation you did last week and perhaps even your music collection folder. The desktop search application must be able to read all of these different file formats and interpret them. It must understand the file system of the computer you are on (easy), the file format of the files it is searching (easy but extensive as some of these file formats may be closed) and then comes of the problem of ranking it. How do you classify as more important - the email sent last week or the document which includes your name in the title? Whilst Google has made it's name on the Page Rank algorithm, there are no links coming into your files that it can readily look at.

Now do the same search on your Intranet. All the same problems of desktop search now exist but now consider this:

1. You are searching multiple file systems, across multiple servers (Windows, Linux, Unix etc) and even potentially, a SAN. Should it include shared PCs?
2. The file format across the network may now be expanded to include databases, email storage systems (such as Exchange or Notes) and also document management systems.
3. The issue of security come to mind. Should I be able to see all of these documents across the network?

Point 3 was raised at a Microsoft event I attended in late 2000 when Share Point Portal Server was launched. An employee did exactly what I have suggested and searched for his name across the Intranet. The search returned a document called "Employee Name Termination Letter Date.doc". The employee was unable to read the document but based on the title, knew that his days at the company were limited.

Whilst there is a lot to be gained from the Search companies looking at each different area of search, the challenges that exist in each are unique. I am not saying that there is not overlap across each sector but many of the lessons learnt are unique to that particular search segment. Personally I feel the area of Enterprise Search is the most complicated, if only for the reasons I have called out.

Perhaps Yahoo! is smart in not expending resources into an area that has many complexities and focusing on its core users - individuals looking to find information on their desktop or on the web.

Posted by Warren Duff at 8:18 AM GMT