Enterprise search differs from web search in some fundamental ways. But there are also similarities. Since we all know how successful web search is, let’s see if there is something to learn by examining the differences.

The most striking similarity is that both solutions involve content produced by a user.

The main differences are the search engine and who controls it, the different types of content sources, and the use of the pagerank algorithm.

Web content and web search

What makes web search so successful?  Web search was revolutionized when Google introduced their web search using the pagerank algorithm. Pagerank uses the natural structure of the world wide web, and assigns high weight to pages with many incoming links. It rests on the assumption that pages with correct and important information will be used as references on other pages. Along with pagerank, there is a large number of other factors used to drive relevancy; content quality, keywords, social media sharing etc. Most of the details are not publicly available.

Content publishers on the world wide web can use the tricks of Search Engine Optimization (SEO) to make sure their content gets optimal visibility on the web. SEO is the art of combining knowledge of two things;

- how web search engines work

- what search terms people use.

Both of these areas involve a lot of guessing. Some information can be found in guides such as the Google Webmaster Guidelines, which explains what a webmaster can do to make sure her web sites are properly indexed. Parts of this read almost like an instruction on how to create a nice school paper: “Create a useful, information-rich site, and write pages that clearly and accurately describe your content.”

By following these guidelines, you are helping web search engines understand your content.

Enterprise content and enterprise search 

Enterprise search is a different story. Content is gathered from different sources, with varying degrees of  structure, and mostly without links that could be used for pagerank.

Content publishers in an enterprise search solution are on their own, with no official guidelines describing the rules to follow to win top ranking on the intranet search. More often than not, nobody knows how the enterprise search engine really works. Compare this to the web search situation, and it should not come as a surprise if enterprise search sucks.

A solution to this dilemma requires taking a step back from the idea that enterprise search is a box that you can plug in to your intranet  and “there was search”.

Search tech guys often pride themselves in the abilities of their search engines, and will rather fix relevancy problems created by bad content by doing tricks on the technical side of things. On the other side of the story, content producers expect search to “just work”, and put all the responsibility on technology and the implementer.

Creating a great enterprise search solution requires cooperation between the makers of content and the makers of search solutions.

Content producers should know how their content will end up in search. They should know what factors affect findability. Search solutions should have documentation targeted towards the end user, which in the enterprise also might be a content producer.

ESO

We can define Enterprise Search Optimization (ESO) as the art of improving Enterprise Search. Where ESO has been applied, we should expect to find a well functioning search solution, where employees and content producers know how to create easily findable content.

Compared to SEO, Enterprise search optimization is a simple procedure, involving little  guesswork in regard to figuring out how the search engine works. It is also a difficult procedure, since ESO needs to be individually tailored and optimized for the specific informational needs for each enterprise.  To develop ESO guidelines, the search technicians need to sit down with the content producers and users to figure out the details of the information model and where the pain of missing information hurts the most.

ESO should result in a list of guidelines, or rules, similar to the lists of SEO. These rules can range from simple and obvious, making sure documents have descriptive titles, correct dates and author, to more complex involving consistent language use, metadata fields for categorization, etc. ESO rules should also explain how structure is imposed on data with less structure.

Recognizing authoritative content is solved in web search engines by using the pagerank algorithm. Enterprise search will rarely be able to use pagerank directly. Authority can often be determined by other means. This can range from simple facts like “This book is the company procedure bible” to “powerpoint is more important than word”.

Optimizing  enterprise search

Enterprise search can suck a little less by applying a customized version of SEO.

One generic tool for getting the right results on top is the “term frequency–inverse document frequency“, or tf-idf for short. It’s a combination of how often a term is mentioned in a document compared to how often it’s mentioned in all of your documents in the index. So, rare terms within the whole index used often in one document makes it a good search result when searching for that term. But most likely, not good enough. You need to figure out what’s the characteristics of your content, and what are the most characteristic use cases and user stories for your users. Only then can you achieve great relevancy, …  I mean get the right result on top of your search result.

Model for relevance tuning

We’ll use our Recipe app as an example…

So, for our food recipe app, we have some obvious content characteristics:

• The more ingredients in-season for one recipe is good. We’re doing an OR-search on all ingredients in-season so this comes out-of-the-box … almost.
• Quite a lot of recipes doesn’t stand the test of time. We know that most of the recipes at oppskrift.klikk.no from 2008 or newer are quite good and have nice photos.
• We’re not sure if we need this, but we know whom of the writers to trust. This may be an overkill when we already have a boosting on newer recipes.

And we know a lot about our users as well:

• Most grown up people in Norway have a job, thus limited time to prepare a meal. This means that recipes that takes shorter preparations should be boosted from Monday through Thursday. The verdict on Friday is still not decided.
• During the weekend people have more time to make dinner. The recipes that takes a short time to prepare most probably cut some corners, and are not that good compared to recipes that takes a little longer time. So for the weekends, we should do a demotion of really quick recipes, at least for dinners.

This is the info we’re going to use to sort our search result. But we have more knowledge about our users that we can use to auto-set filters for certain times of the day:

• Most work days, people don’t plan a breakfast meal or lunch. The whole day we can auto-set the main “course” filter.
• During the weekend, people may also plan a lunch. We’ve decided to auto-set the “light meal”-filter during weekends up until lunch time. After that the “main course” filter is auto-set. We’ll log if the first thing our users do is to set another filter.
• On Friday and Saturday a lot of Norwegians drink beer, wine or liquor. After some hours of drinking, they get hungry. Maybe we should have an “afterparty, quick and greasy and tasty-meal”-filter auto-set for late Fridays and Saturdays?

So, what’s the filters we’ve decided on:

• Light meals
• Starters
• Main courses
• Deserts
• … and maybe the Afterparty-thingy

What do you think?

Sounds nice? This is work in progress, so check back every now and then for new blog posts.

1. Learn more about search, both tech and UX.
2. Show that a nice search user experience doesn’t need a search input box.
3. Show a search that is light weight.
4. Generally, build something nice, quickly, to show off.

“But what is it?” you ask again. The second answer is that it’s a search application for recipes containing the most in-season vegetables, fruits, mushrooms, fish, shellfish, poultry and meat. So, a search without a search box giving you the recipes that will make you pick the freshest, most tasty food available at any given time of the year. In Norway, you can get strawberries the whole year around, but it’s only a month or two during the summer that they taste really good. and are cheap. This goes for a lot of different foods.

We’ll blog every step of the journey. First small hackathon is this Friday, January 7th. To test our ideas we’ve already crawled food recipes from Klikk.no, and transformed them into JSON-format by using Forage Document Processor from Fergus McDowell’s excellent Forage Search Engine.

Next step is to get an Amazon EC2-server up and running, install Elasticsearch, and feed it recipes in JSON-format. We’ll also make some rough mockups of how the user interface is going to be, and maybe, hopefully get a simple version of the Seasonal Food Recipe Web Application up and running. If we after a while get enough traffic, we’ll try to do some machine learning. And we’ll definitely test out different ways of doing relevancy tuning.

Sounds nice? This is work in progress, so check back every now and then for new blog posts.