Search Nuggets » enterprise search

Search: better user experience with one line of JavaScript

Espen Klem — Mon, 18 May 2015 13:59:01 +0000

What’s the cheapest trick you can do to get a better user experience on your search solution, and make your users do better search queries?

Add a small line of JavaScript in your template’s document ready function:

$("#MySearchBox").focus();

This will do two things for the user:

It’ll be easier to see the search box .
The user can start typing without having to click inside the search box.

Next issue is that most intranet and websites are more than just a search solution. Maybe you don’t want that much attention on the search box on your homepage. The solution is then to do this on your search result page.

This will make it easier for your users to enhance their search query when they’re not happy with the search result at hand.

Do you have any other examples on other quick fixes that could make an even better user experience for your search solution?

Ny versjon av Comperio FRONT.NET

Christoffer Vig — Wed, 13 May 2015 10:24:54 +0000

Comperio har gjennom tidenes løp levert over 100 søkeprosjekter. Tankegods, svette og erfaringer hentet fra dette arbeidet har krystallisert seg inn i vår egentuviklede programvare for søkeapplikasjoner: FRONT. Tidligere i vår lanserte vi versjon 5 av Java-FRONT, denne gang er det den noe yngre fetteren Comperio FRONT.NET som har fått ligge på operasjonsbordet. Hovedtrekkene i den nye versjonen er nye søkeadaptere, forbedret stabilitet og ytelse, samt forbedret logging.

Mellomvare for søk

FRONT.NET opererer som mellomvare, og lar deg konfigurere forretningslogikk for søk uavhengig av både søkemotor og presentasjon. FRONT.NET er laget for å kunne hente og sette sammen informasjon fra ulike kilder, og kan gjerne kalles en søkeorkestrator.

FRONT.NET lar deg skille mellom forretningslogikk og applikasjonslogikk. Applikasjoner som trenger søkefunksjonalitet trenger ikke bry seg med kompliserte søkeuttrykk, men sender simpelthen spørreord over til FRONT.NET. Trenger du å avgrense søket, kan du sende med filter, som for eksempel brukerinformasjon, sted, avdeling, eller lignende. De komplekse spørringene tar FRONT seg av.

Søkemotoruavhengighet

FRONT.NET tilbyr et generelt format for spørringer, og søkeresultater. Dataformatet fra FRONT er det samme, uavhengig av om motoren i bakkant er SharePoint, ESP, eller Solr. FRONT.NET har i dag adaptere for Fast ESP, SharePoint 2010 og 2013, Elasticsearch, Solr og Google Search Appliance. Dette gjør det enkelt å sette sammen resultater fra ulike søkemotorer. Dersom du ønsker å bytte ut søkemotoren trenger det ikke innebære endringer i din applikasjon, da det kun er snakk om å bytte ut søkeadapter i FRONT.NET. Nye adaptere utvikles så snart vi ser behovet melde seg.

Elasticsearch adapter

Elasticsearch er en søkemotor i stor vekst. Til utvikling av Elasticsearch adapteret har vi kunnet dra nytte av NEST, den offisielle .NET klienten for Elasticsearch. Elasticsearch har enorm fleksibilitet i forhold til hvordan spørringer kan uttrykkes, med mulighet for nestede boolske uttrykk og dynamiske ranking-funksjoner. I utvikling av adapteret har vi valgt å minimere kompleksiteten i FRONT ved å delegere disse mulighetene inn i Elasticsearch via søkemaler (search templates). Dette ivaretar fleksibiliteten, samtidig som APIer og programmeringsgrensesnittene er beholdt.

Google Search Appliance Adapter

Comperio ble ifjor partner med Google, og vi har nå utviklet FRONT.NET adapter for Googles intranett søkemotor Google Search Appliance, eller bare GSA for kort. GSA tilbyr enkel integrasjon mot en rekke ulike kilder, søkegrensesnittet er enkelt og forholde seg til og adapteret har støtte for alle vanlige søkeoperasjoner.

Logging

For å kunne utvikle en god søkeløsning er det avgjørende at man har tilgang til gode søkelogger som avslører hvordan søkeapplikasjon brukes.
FRONT.NET har nylig fått funksjonalitet for å kunne logge direkte til Logstash. Logstash kombinert med Elasticsearch og Kibana gir deg et kraftig verktøy for dataanalyse.

FRONTD

Versjon 5 av FRONT.NET kjører som en frittstående tjeneste i Windows.
Tidligere versjoner opererte som web applikasjon under IIS (Internet information server), men vi ser at når vi kjører frittstående oppnås forenklet administrasjon, samt forbedret stabilitet og ytelse.

Microsoft, .NET og veien videre

Microsoft og .NET verdenen er under rivende utvikling for tiden, ikke minst gjennom Microsoft sin nye og kjært velkomne åpning mot open source. Vi liker veldig godt ideen om kryssplattform .NET, og neste versjon av FRONT.NET vil forhåpentligvis kjøre like bra på OS X og Linux som på Microsoft.

Public 360 connector for SharePoint søk

Pål Christian Kjølberg — Mon, 09 Feb 2015 12:37:50 +0000

Software Innovation sine systemer for dokumenthåndtering, Public 360 og Business 360, er brukt av mange offentlige og private aktører både i Norge og Sverige.

Comperio gjør det enkelt å finne riktig dokument selv om du ikke er godt kjent med Public 360. Ved å gjøre innholdet i Public 360 tilgjengelig i en søkeindeks kan man få søketreff på tvers av kilder. SharePoint 2013 tilbyr en slik søkeindeks. Denne vil kunne gi presise treff fordi spørringen henter treff basert på data både fra dokumentet og metadata til dokumentet. På den måten kan man utforske hva som finnes av innhold og historikk knyttet til et tema eller en sak.
For å få tilgang på dokumentene i Public 360 må man koble kildesystemet sammen med søkemotoren med en konnektor.

Comperio laget i første omgang en konnektor for å kunne indeksere innhold fra Public 360 basert på Fast Search Server for SharePoint 2010. Siden har vi laget en ny versjon for Microsoft SharePoint 2013.
Når vi laget konnektoren så bruke vi Microsofts rammeverk for å lage tilpassede BCS (Business Connectivity Services) konnektorer til SharePoint 2013.

Så, hvordan fungerer konnektoren for indeksering av innhold?

FileFinder web servicen er endepunktet i Public 360, og det brukes til å hente Public 360 dokumenter. Fra denne web servicen får vi all informasjon om hvert dokument, som egenskaper, metadata og rettighetsinformasjon. Videre blir hvert dokument og dets egenskaper plukket opp av SharePoint 2013, hvor dokumentet crawles, prosesseres og lagres i søkeindeksen.

Og hvordan fungerer konnektoren ved søk utført av en bruker?

Når en bruker gjør et søk i søkeboksen, så legges det til hva slags tilganger brukeren har på spørringen. Disse tilgangene hentes fra Public 360 ved hjelp av en SharePoint security trimmer som settes opp for Public 360. Dette fører til at kun dokumenter som brukeren har rettigheter til blir returnert som en del av søkeresultatet.

Ta kontakt for å høre mer om Public 360 og SharePoint.

Enterprise Search Optimization (ESO)

Christoffer Vig — Sat, 10 Jan 2015 11:45:54 +0000

So, you got your enterprise search engine, but still can’t find what you’re looking for? It’s time to stop your sobbing and learn to play the exciting game of Enterprise Search Optimization (ESO).

Enterprise search differs from web search in some fundamental ways. But there are also similarities. Since we all know how successful web search is, let’s see if there is something to learn by examining the differences.

	Web search	Enterprise search
Search Engine	Google, Bing, Baidu…	SharePoint, Elasticsearch, Solr, Virtualworks, Autonomy, GSA…
Sources	Web pages, web applications (++)	databases, file shares, intranet, web pages, email, SAP, CRM…
Content ambitions	everything	limited
Authority ranking	Pagerank	custom
Control of search engine	web search company	tech department, power users
Control of content	user	user
Writer/reader ratio	low	high

The most striking similarity is that both solutions involve content produced by a user.

The main differences are the search engine and who controls it, the different types of content sources, and the use of the pagerank algorithm.

Web content and web search

What makes web search so successful? Web search was revolutionized when Google introduced their web search using the pagerank algorithm. Pagerank uses the natural structure of the world wide web, and assigns high weight to pages with many incoming links. It rests on the assumption that pages with correct and important information will be used as references on other pages. Along with pagerank, there is a large number of other factors used to drive relevancy; content quality, keywords, social media sharing etc. Most of the details are not publicly available.

Content publishers on the world wide web can use the tricks of Search Engine Optimization (SEO) to make sure their content gets optimal visibility on the web. SEO is the art of combining knowledge of two things;

- how web search engines work

- what search terms people use.

Both of these areas involve a lot of guessing. Some information can be found in guides such as the Google Webmaster Guidelines, which explains what a webmaster can do to make sure her web sites are properly indexed. Parts of this read almost like an instruction on how to create a nice school paper: “Create a useful, information-rich site, and write pages that clearly and accurately describe your content.”

By following these guidelines, you are helping web search engines understand your content.

Enterprise content and enterprise search

Enterprise search is a different story. Content is gathered from different sources, with varying degrees of structure, and mostly without links that could be used for pagerank.

Content publishers in an enterprise search solution are on their own, with no official guidelines describing the rules to follow to win top ranking on the intranet search. More often than not, nobody knows how the enterprise search engine really works. Compare this to the web search situation, and it should not come as a surprise if enterprise search sucks.

A solution to this dilemma requires taking a step back from the idea that enterprise search is a box that you can plug in to your intranet and “there was search”.

Search tech guys often pride themselves in the abilities of their search engines, and will rather fix relevancy problems created by bad content by doing tricks on the technical side of things. On the other side of the story, content producers expect search to “just work”, and put all the responsibility on technology and the implementer.

Creating a great enterprise search solution requires cooperation between the makers of content and the makers of search solutions.

Content producers should know how their content will end up in search. They should know what factors affect findability. Search solutions should have documentation targeted towards the end user, which in the enterprise also might be a content producer.

ESO

We can define Enterprise Search Optimization (ESO) as the art of improving Enterprise Search. Where ESO has been applied, we should expect to find a well functioning search solution, where employees and content producers know how to create easily findable content.

Compared to SEO, Enterprise search optimization is a simple procedure, involving little guesswork in regard to figuring out how the search engine works. It is also a difficult procedure, since ESO needs to be individually tailored and optimized for the specific informational needs for each enterprise. To develop ESO guidelines, the search technicians need to sit down with the content producers and users to figure out the details of the information model and where the pain of missing information hurts the most.

ESO should result in a list of guidelines, or rules, similar to the lists of SEO. These rules can range from simple and obvious, making sure documents have descriptive titles, correct dates and author, to more complex involving consistent language use, metadata fields for categorization, etc. ESO rules should also explain how structure is imposed on data with less structure.

Recognizing authoritative content is solved in web search engines by using the pagerank algorithm. Enterprise search will rarely be able to use pagerank directly. Authority can often be determined by other means. This can range from simple facts like “This book is the company procedure bible” to “powerpoint is more important than word”.

Optimizing enterprise search

Enterprise search can suck a little less by applying a customized version of SEO.

ComperioFrokost 11. juni

Trond Renshusløkken — Wed, 11 Jun 2014 15:43:10 +0000

Takk til alle 55 deltakerne på vårt frokostseminar på THE THIEF onsdag morgen!

Noen utvalgte bilder fra arrangementet:

Comperio Search London Breakfast Seminar 14th March

Job Maelane — Mon, 17 Mar 2014 13:07:27 +0000

Comperio Search would like to thank all the guests who attended the Breakfast seminar last Friday, the 14th of March 2014 in London.

Download the presentations:

Leveraging Search Based Applications on new Intranet
(David Wright, Lead SharePoint Architect, Coutts)
Searching Internally Using Enterprise Social Networks
(Zane Freame, Technical Sales Professional, Microsoft UK)
Enterprise Search Strategy by Comperio Search
(Job Maelane, John Thompson, Comperio Search)

Some pictures from the event…

Comperio Search welcomes its guests...

Zane Freame, Technical Sales Professional, Microsoft UK

David Wright, Lead SharePoint Architect, Coutts

Trond Renshuslokken, Comperio CEO

ComperioFrokost 12. mars

Trond Renshusløkken — Thu, 13 Mar 2014 06:28:25 +0000

Takk til alle 65 deltakerne på vårt frokostseminar på Continental onsdag morgen!

Last ned presentasjonene:

Søkemarkedet i 2014 (PDF)

Trond Renshusløkken og Espen Klem, Comperio
Bruk av søketeknologi i rettsdata.no (PDF)

Kristian Skeie, Forlagsdirektør, Gyldendal Rettsdata
Om Postens Tilbudssøk.no (PDF)

Live Marstein, Prosjektleder, Posten Norge
Hvordan kan du være innovativ i din virksomhet med Google-teknologi? (kommer)

Christopher Conradi, Enterprise Sales Manager, Google

Noen utvalgte bilder fra arrangementet:

Takket være oppussing på Continental fikk alle frokostdeltakerne starte dagen med litt trappetrim.

Daglig leder i Comperio Trond Renshusløkken ønsker velkommen.

Trond Renshusløkken, Comperio

Espen Klem, Comperio

Kristian Skeie, Gyldendal Rettsdata

Live Marstein, Bring/Posten

Christopher Conradi

Følg oss på Twitter så du ikke går glipp av neste frokostseminar.

Dynamic search ranking using Elasticsearch, Neo4j and Piwik

Christian Rieck — Wed, 05 Feb 2014 14:49:52 +0000

Getting the correct result at the top of your search results isn’t easy. Anyone working within search quickly realizes this. Tuning the underlying ranking model is a job that just doesn’t end. There is an entire profession about search engine optimization, making sure your site gets as high as possible on Google (and Bing, I guess). If it is not the top result on Google, it is somehow your fault and not Google’s.

Nobody optimizes for an internal enterprise search solution

If your document is not the top result in the internal search solution it is somehow the search engine’s fault, not yours. There is no link cardinality on a file system. All the metadata is wrong and the document your user is trying to find doesn’t even contain the words the user remembers it to contain; the end result being that the target document is not found. As a result of this, trust in the enterprise search diminishes and soon you are left without users. Let’s see how we can use Piwik, neo4j and Elasticsearch to remedy this. (Yes, you can use Solr if you want).

This post is made up of three parts. First I’ll talk about gathering the data necessary. Then we’ll tackle getting the ‘right’ documents at the top of your search and lastly we’ll see if we can expand documents with words your users recalls them by, but are not part of the documents themselves. The journey will be based on the work performed on Comperio’s internal search, at the moment implemented on an old Fast ESP installation.

Gathering data

First you need to know what your users are searching for and what they end up clicking on. We use Piwik, an open source web analytics platform, for this. Seeing the searches, modifications to the searches and if they ended up clicking on anything that they thought was exciting. For a while we only used this for statistics since Piwik offered better insight than the built in query statistics in Fast ESP. Here is an example of one search session:

We see a user entering the site, querying ‘rank order words’ and clicking on a document. Then the same search is executed again. It is reasonable to conclude the clicked document did not contain the wanted information. Lastly ‘boost position term’ is searched. Sadly the session does not end with a click so I guess our search couldn’t deliver. :( [1]

In their current form, the statistics aren’t very useful. But what were to happen if we took these chains of activities and created a graph? We used neo4j for this. A small Java program was written to download the Piwik-history as an XML-file and insert it into a newly created neo4j database.

The nodes are either the start of a session, a search or a document. They are linked by relationships such as CLICKED, SEARCHED, RETURNED_FROM. Since a neo4j database isn’t very screen shot friendly, here is a part of the graph as rendered by Gephi:

We see someone looking for help with Chinese query suggestions. S361 marks the beginning of this session and the first search term was ‘chinese’. They then clicked a link for an internal mail archive before refining their search to ‘chinese als’ and so forth. Links that show when a user back tracked are not shown. That was an isolated little island. The more central documents and search terms at your company will create bigger webs.

Seeing your search history organized like this should give an urge to dive in and explore. It is really interesting, fun and recommended!

Finding popular documents

The simplest way of finding the popular documents is to track search term -> clicks directly. It is also the most common way of doing it. That wouldn’t utilize our fancy new graph now, would it? Since we can do queries against the database let’s get all search sessions of 8 or less actions that resulted in a click on document X:

Page Break(Small disclaimer: As my neo4j skills are very rudimentary there might be more efficient ways of doing this.)

Now we iterate over all sessions and give a score to each search term. The closer it is to the clicked document, the higher score it gets. Sum the score across all sessions. After doing that you get a score indicating how ‘close’ a search term is to any document. This is example data for the single-word search term ‘vpn’:

When the score passes a threshold we add the search-document pair to an Elasticsearch index. For every search executed at our search we first check Elasticsearch to see if the term is boosted. For ‘vpn’ the search logs state

We can see how three documents are boosted for ‘vpn’. (By choice we only boost the top three). Using Fast ESP we wrap the original query with boosts for those specific documents.

In FAST ESP, as well as in Sharepoint Search 2013 the beloved xrank-operator is your friend. In a Lucene based search application use boost queries for this.

The search result returns the popular hits (only one shown here) at the top

The ugly star and cheesy feedback is me trying to tell the users rather bluntly that things happened behind the scene and that their actions will affect future searches. Currently there is no way of giving negative feedback to say ‘no, this is actually not a good hit’. Oh well.

As a bonus all terms that results in boosted documents are, as far as we know, smart things to search for and free of spelling errors. Therefor all such terms are added to a second Elasticsearch index we base our query completion on. (As a side note – if misspelled terms appear often enough to overcome the threshold for them to be taken into account, they could be part of your organization’s tribal language. If the users choose to spell the term “definately” so often that it “makes the cut” then the system should adapt to that. )

Expanding documents to increase recall

Often a user thinks of one document and searches for what, to them, identifies the document. That term might or might not be present in the document itself. If it doesn’t the document is not returned and the user becomes sad. Hopefully they alter their search and continue to look. Should they end up at their document we have the tools needed to remedy the situation. Here is a concrete example:

Here we can see that the node marked 1 might be tagged with ‘sort order refiner entries’ or at least ‘refiner’, a term used twice when trying to find this document. (As an interesting side note, if you observe a lot of ‘sort X’ followed by ‘sort Y’ you might consider adding a synonym between X and Y.) If a term or phrase is used often enough across different sessions we save this to an Elasticsearch index. Each time a document is indexed we look up the document in our index and add any popular search terms to a low ranking field. This guarantees a recall of the document but it will not automatically top the results for those queries. This is a two-step process. If your search engine supports partial updates of documents, go with that.

Before adding the last step we noticed that for some searches we boosted documents that didn’t get recalled and thus were never displayed to the user even though we knew it was a good hit!

Closing words

As a first step towards dynamic ranking this has shown good results. As long as your search engine supports query time boosting you can implement this.

By the way

It should be noted that SharePoint will actually do some of this for you. It comes with an interface meant to be used by an end user (as opposed to all search engines I’ve seen) and the UI contains the event listeners on all links, tracking what you do. This is fed into a database and the data does affect ranking. As far as I know only the last search term before a click is associated with the clicked link.

[1] One scenario that Piwik and click tracking does not pick up is if the sought information is found in the returned teasers. Search sessions that don’t end in a click might in fact have a happy ending.

Big Data and Enterprise Search

John Thompson — Wed, 27 Feb 2013 13:32:02 +0000

There have been a number of reports and papers issued recently on Big Data including:

Forrester reviewing the Big Data solutions of 2013
The Economist talks about what is Big Data and how can it be used
Wall Street Journal talk about what is next for Big Data
The Sunday Times reviewing how Big Data has helped various companies
Martin White discusses if there is a need for enterprise search whilst Big Data lives
Stephen Arnold discusses his Big Data trends for 2013
Mike Walsh reviews Big Data strategies for 2013

As well as software vendors pushing their appetite for Big Data, either via their websites e.g. Microsoft and their Big Data Week or via taking adverts in the UK national press i.e. IBM running a number of full page adverts in The Times, week commencing Monday 18^th February 2013.

From reading these reports and papers, what actually is Big Data? Is Big Data a hype? Is it only relevant for a small number of very large organisations where the volume, rate of change, variety and worth of this data is highly relevant? It is very hard to answer these questions – what may be Big Data to you may not be Big Data to me.

Big Data to me is being able to capture data, whether it is structured, semi-structured or totally unstructured, store it, interpret it, and leverage it to provide insights in order to help the business.

So how does Big Data and enterprise search co-exist? Can traditional search tools work as the “gateway” to explore Big Data by, for instance, preparing the data to help in creating the predictive model?

Based upon the Forrester report mentioned above, SAS and IBM are the leaders in the Big Data space with a large number of tools available to process and analyse the data. As the first part of any analysis is preparing the data, and with a large proportion of the data being unstructured, could enterprise search use its distinctive capabilities of pre-processing large amounts of both structured and unstructured content – I think not. Currently, enterprise search tools do not have the capability to traverse some a large amount of data in a timely manner in order to try and produce a smaller, relevant content set without using a very large amount of hardware. The structured data that exists may already be well organised but unstructured content is another matter and trying to interpret meaning between structured and unstructured content can be very complex.

So, if enterprise search cannot help directly with preparing the data for analysis, where can search help? Frost and Sullivan forecast the global enterprise search market to be US$4.68bn by 2019. Search has the capability to bring a wide variety of different content sources together and produce meaning. Search queries, from simple to complex, can then be run against the search index returning, hopefully the most relevant content based upon the search query terms entered. But, for the most part, this won’t give the answers to the Big Data questions e.g. – helping to uncover the answers to making the best use of the data available.

However, Search Based Applications (SBA), as defined by Wikipedia , “use semantic technologies to aggregate, normalize and classify unstructured, semi-structured and/or structured content across multiple repositories, and employ natural language technologies for accessing the aggregated information.”, can be built to slice and dice the information in the search index on-the-fly – isn’t this close to what the Big Data engines are trying to achieve. There are a number of search related companies building SBAs which look to build insight in the realms of data that organisations amass.

There are obviously limitations to what search engines can do in terms of the size of the data sets – that’s why it’s called Big Data. However, there must be a reason why companies like IBM purchased Vivisimo in 2012 or Oracle bought Endeca in 2011, with both companies looking to capitalize on the capabilities both Vivisimo and Endeca offered in terms of unlocking the structured and unstructured content within organisations.

Quote from Oracle on the Endeca acquisition, “Oracle with Endeca plans to create a comprehensive technology platform to process, store, manage, search and analyze structured and unstructured information together enabling businesses to make stronger and more profitable decisions” – search and Big Data complementing each other – I think so.

Finally, the latest information from Gartner indicates “Big Data is forecast to drive $34 billion of IT spending in 2013 and create 4.4 million IT jobs by 2015, but it is currently still a solution looking for a problem”.

SharePoint conference 2012 keynote: raising the bar for Enterprise Search

Marcus Johansson — Mon, 26 Nov 2012 20:42:16 +0000

A little after 8:30 this morning, Jared Spataro, Senior Director of SharePoint, opened this year’s SharePoint Conference (SPC) to the sound of 10,000 SharePoint geeks simultaneously tapping away on their devices, furiously making #SPC12 the trending topic on Twitter.

The release of SharePoint 2013 is touted by Microsoft as a bridge from the past to the future; a pivotal shift from server- and wave-based releases to a rapid release cycle, combining the muscles from the cloud, fresh design principles and Enterprise Social technology. They’re proud, and they should be.

In the keynote, Microsoft also revealed some of the rationale behind the recent acquisition of Yammer, seeing how it complements SharePoint’s strong document-management skills with Yammer’s deep expertise in making the enterprise more social.

Jeff Teper, another high-profile SharePoint executive, told the crowds that over the last three years, Microsoft’s engineering team has centralized their effort around three important pillars: Experiences, Innovation and Ecosystem. This shows off in a dramatically improved user experience, better development tools and a completely revamped application model paving the way for us partner companies to more rapidly and with more flexibility deploy innovative applications in the cloud.

Another consistent theme throughout the keynote, was how Search now has a bigger role across SharePoint and throughout the Microsoft stack. FAST is now fully integrated, and the new search core drives both content and recommendations throughout the suite.

We now have a much better baseline when implementing Search technology in the enterprise. Hopefully this is finally the time when custom implementations of Enterprise Search will stop ending up as just a search box, and instead unleash the potential of Your organization’s accumulated knowledge and tackle today’s overwhelming growth of information.