A few years ago Comperio launched a nifty service for Posten Norge, Norway’s postal service. Through the service, retail companies can upload their catalogues and seasonal flyers to make the products listed within searchable. Although the catalogue handling and processing is also very interesting, we’re going to focus on the search side of things in this post. As Comperio has a long relationship and a great deal of experience with FAST ESP, this first iteration of Posten’s Tilbudssok used it as the search backend. It also incorporated Comperio Front, our search middleware product, which recently had a big release. .

Newer is better

Unfortunately, FAST ESP is getting on a bit and as a result Tilbudssok has been limited by what we can coax out of it. To ensure we provide the best possible search solution we decided it was time to upgrade and chose Elasticsearch as the best candidate. If you are unfamiliar with Elasticsearch, take a moment to browse our other blog posts on the subject. The resulting project had three main requirements:

• Replace Fast ESP with Elasticsearch while otherwise maintaining as much of the existing architecture as possible
• Add geodata to products such that a user could find the nearest store where they were available
• Setup sexy log analysis with Logstash and Kibana

Data Sources, Ingestion and Processing

The data source for the search system is a MySQL database populated with catalogue and product data. A separate Comperio system generates this data when Posten’s customers upload PDFs of their brochures i.e. we also fully own the entire data generation process.

The FAST ESP based solution made use of FAST’s JDBC connector to feed data directly to the search index. Inspired by Christoffers blog post, we made use of the JDBC plugin for Elasticsearch. This allowed us to use the same SQL statements to feed Elasticsearch. It took us no more than a couple of hours, including some time wrestling with field mappings, to populate our Elasticsearch index with the same data as the FAST one.

We then needed to add store geodata to the index. As mentioned earlier, we completely own the data flow. We simply extended our existing catalogue/product uploader system to include a store uploader service. Google’s geocoder handled converted addresses to coordinates for use with Elasticsearch’s geo distance sorting. We now had store data in our database. An extra JDBC river and another round of mapping wrestling got that same data into the Elasticsearch index.

Our approach

Before the conversion to Elasticsearch, the Posten system architecture was typical of most Comperio projects. Users interact with a Java based frontend web application. This in turn sends queries to Comperio’s search abstraction layer, Comperio Front. This formats requests such that the system’s search engine, in our case FAST ESP, can understand them. Upon receiving a response from the search engine, Front then formats it into a frontend friendly format i.e. JSON or XML depending on developer preference.

Unfortunately, when we started the project, Front’s Elasticsearch adapter was still a bit immature. It also felt a bit over kill to include it when Elasticsearch has such a robust Java API already. I saw an opportunity to reduce the system’s complexity and learn more about interacting with Elasticsearch’s Java API and took it. With what I learnt, we could later beef up Front’s Elasticsearch adapter for future projects.

As a side note, we briefly flirted with the idea of replacing the entire frontend with a hipstery Javascript/Node.js ecosystem. It was trivial to throw together a working system very quickly but in the interest of maintaining existing architecture and trying to keep project run time down we opted to stick with the existing Java based MVC framework.

After a few rounds of Googling, struggling with documentation and finally simply diving into the code, I was able to piece together the bits of the Elasticsearch Java API puzzle. It is a joy to work with! There are builder classes for pretty much everything. All of our queries start with a basic SearchRequestBuilder. Depending on the scenario, we can then modify this SRB with various flavours of QueryBuilders, FilterBuilders, SortBuilders and AggregationBuilders to handle every potential use case. Here is a greatly simplified example of a filtered search with aggregates:

Logstash and Kibana

With our Elasticsearch based system up ready to roll, the next step was to fulfil our sexy query logging project requirement. This raised an interesting question. Where are the query logs? As it turns out, (please contact us if we’re wrong), the only query logging available is something called slow logging. It is a shard level log where you can set thresholds for the query or fetch phase of the execution. We found this log severely lacking in basic details such as hit count and actual query parameters. It seemed like we could only track query time and the query string.

Rather than fight with this slow log, we implemented our own custom logger in our web app to log salient parts of the search request and response. To make our lives easier everything is logged as JSON. This makes hooking up with Logstash trivial, as our logstash config reveals:

Kibana 4, the latest version of Elastic’s log visualisation suite, was released in February, around the same time as we were wrapping up our logging logic. We had been planning on using Kibana 3, but this was a perfect opportunity to learn how to use version 4 and create some awesome dashboards for our customer:

Kibana 4 is wonderful to work with and will generate so much extra value for Posten and their customers.

Conclusion

• Although the Elasticsearch Java API itself is well rounded and complete, its documentation can be a bit frustrating. But this is why we write blog posts to share our experiences!
• Once we got past the initial learning curve, we were able to create an awesome Elasticsearch Java API toolbox
• We were severely disappointed with the built in query logging. I hope to extract our custom logger and make it more generic so everyone else can use it too.
• The Google Maps API is fun and super easy to work with

Rivers as a data ingestion tool have long been marked for deprecation. When we next want to upgrade our Elasticsearch version we will need to replace them entirely with some other tool. Although Logstash is touted as Elasticsearch’s main equivalent of a connector framework, it currently lacks classic Enterprise search data source connectors. Apache Manifold is a mature open source connector framework that would cover our needs. The latest release has not been tested with the latest version of Elasticsearch, but it supports versions 1.1-3.

Once the solution goes live, during April, Kibana will really come into its own as we get more and more data.

Takk til alle som valgte å bruke morgenen på å høre et par av våre kunder komme med konkrete erfaringer fra sine prosjekter, og få litt inspirasjon til hvordan håndtere sine egne informasjonsutfordringer. Det var mange blide ansikt og gode tilbakemeldinger etter seminaret.

Agenda for seminaret finnes her

Se presentasjonene fra foredragene

Her følger noen utvalgte bilder fra arrangementet

Introduksjon og velkommen.

Helge Blindheim forklarte entusiastisk om Helsedirektoratets tanker og visjoner, og hvordan Sharepoint søk skulle bidra til å nå visjonen om skreddersydd og sikker helseinformasjon til alle.

Thomas Evensen og Jon Solberg forklarte litt om hvor viktig det var for Forskningsrådet at de på en enkel måte kan synliggjøre hvor alle deres tildelinger går. Spesielt morsomt var kanskje at man på en enkel måte kunne se hvor mye en Nobelpris kostet, ved å søke seg fram til og se på prosjektene ekteparet Moser hadde vært prosjektledere på.

Jon Solberg forklarer litt om oppbygningen av Prosjektbanken, og viste hvordan man for eksempel kunne se hvor mange kroner en Nobelpris kostet gjennom å finne alle prosjektene til ekteparet Moser.

Comperios egen André Lynum forklarte hvordan man kan hente ut samhandlingsmønster ved hjelp av grafteknologi generelt og Neo4J spesielt.

André Lynum beskriver hvordan man kan tolke toppolitikeres twitter-aktivitet og hvordan de omtaler hverandre ved hjelp av grafteknologi.

Last ned presentasjonene

Helge T. Blindheim: Helsenorge.no (PDF)

Andre Lynum: Grafteknologi – samhandling, søk, analyse (PDF)

Noen utvalgte bilder fra arrangementet:

Ole-Kristian Villabø ønsker velkommen.

Ole-Kristian Villabø om Comperio sine 10 år

Per Ejnar Thomsen fra TINE om deres virksomhetssøk.

Kjell Skognes om SINTEFs søkeløsning SIPOK

Fagleder Hans Terje Bakke om blant annet Elasticsearch og Kibana.

For eksempel kan jeg hjelpe et selskap med å lage bedre søk for å finne informasjon internt i organisasjonens datasystem. Det første jeg gjør er å sortere filene. Jeg «tagger» filene med relevant informasjon, som for eksempel hvem som har skrevet den eller hva slags prosjekt den tilhører. Problemet er ofte at man setter inn et ord for å beskrive en fil, og når man skal søke det opp bruker man et synonym. Slik informasjon er ikke alltid i filene, derfor må de behandles før de kan søkes opp.På den andre siden har du brukeren som vil finne filene.

Akkurat nå jobber jeg med å gjøre søkene bedre på brukersiden. Noen ganger vet ikke brukeren hva som er det beste nøkkelordet eller søkeordet for det hun eller han søker på. Da kan det for eksempel være nyttig å finne ut hvem som kan noe om temaet. Det vil si at vi kan linke søkeord med eksperter innenfor bedriften. Egentlig gjør vi søket lettere ved å tilgjengeliggjøre informasjon.

Les mer

Before the presentation we had a stand and met many computer science students who asked questions about everything from a simple “What’s up?”-question to more complex ones like “What algorithms do you use to calculate ranking in a search result?”.

Later on, we kept a two-hour session for the students. First, we had a presentation about how search technology works showing different types of search scenarios and how it is to work with search solutions. Also, we had one of our interns from this year, Øyvind Wedøe to present his summer project about how it was to make a 100% search-driven website (beta.comperio.no). In the second session, we gave the students a short introduction about the powerful open-source search engine, Elasticsearch, along with a technical workshop where we showed the students how to set up an index, feed it with data and create simple queries against it (you can read more about it here).

After the presentation, we went to a restaurant with the students where we got to know them more and had a good time.

Comperio consultant, Marcus Johansson, co-authors ‘Working with Microsoft FAST Search Server 2010 for SharePoint’.

As most search techies will testify, the life of a consultant working on complex enterprise search projects with large customers, can be quite a demanding one. To commit to investing personal time, outside of all this project work is, to put pen to paper and document a series of learnings and tips in a structured and user-friendly format, is no mean feat. So on this note, we salute Comperio consultant Marcus Johansson and congratulate him on the release of his first book earlier this month - Working with Microsoft FAST Search Server 2010 for SharePoint.

Marcus co-authored the book with Mikael Svenson and Robert Piddocke, who are also highly regarded contributors to the search community. Here, in Marcus’ own words, is some background on the book:

The book is split into two parts: the first one explains what you’ll need to know to deploy and administrate a solution. The second one is targeting developers who want to build their own search solutions on top of FAST Search for SharePoint. Although we expect most readers to have a SharePoint background, people who’s worked with FAST technology in previous incarnations will hopefully feel at home as well.

Recognizing  FAST Search for SharePoint’s popularity, and seeing that the product is many people’s first exposure of Enterprise Search technology, the first two chapters introduce the reader to important concepts and terminology. We felt it was critical to get this backdrop right as there are vastly different data and user experience opportunities (and challenges) in Search technology, than in e.g. a database-driven solution.

The next few chapters  target the IT Pro audience and deal with such things as architecture, scale-out, deployment and security. Scaling, in particular, is covered in depth seeing how FAST Search for SharePoint is often used for top-tier solutions both in  query load and content volume. Additionally, day-to-day operations and the various methods of maintenance is covered in detail. We show you how to interact with and follow-up the system through the SharePoint GUI, but also how to work with the solutions through PowerShell and, for those adventurous people who wants to truly integrate the platform in their application environments, how you can integrate against FAST Search for SharePoint’s native APIs even for administrational and operational tasks.

The second part of the book is a good fit for application developers who want to create their own search solutions on top of FAST Search for SharePoint, whether that means extending a default FAST Search Center or completely build their own search-driven application truly taking advantage of the powerful platform. Doing so, developers will realize an Enterprise Search platform, such as FAST Search for SharePoint, relies not only on a competent and flexible index, but also on a framework for advanced content processing and the possibilities to query into the data using a rich query language.

Finally, the book is wrapped up with a tutorial-like section on how to resolve common problems and how to attack frequent development scenarios.

At Comperio, we see how more and more of our clients realize the potential in not only implementing a global search experience in their enterprise, but to deploy targeted search-driven applications across their organizations. One user group might need a mash-up of data aggregated from several internal content sources, and perhaps another group needs to trigger a certain function that he or she perceives as a single operation, but in reality hits 2-3 different source systems in the backend. A search-driven interface is the very sweet spot for building such applications, and  I’m hopeful this book will give you the tools of the trade to implement them in reality.

Happy reading!

In recognition of Marcus’ achievement, the team here in Comperio have given him temporary clearance to use lines like ‘As I say in my book, Working with Microsoft FAST Search Server 2010 for SharePoint, …’ in internal meetings and possibly even at customer presentations.

The book is now available for purchase on Amazon - or you can win one of three free copies being offered by Comperio, by being one of the first to tweet this post mentioning @comperiosearch.

Dr. Olstad speaks on how Norway is a hotbed for search technology development and this search ecosystem has it’s source at NTNU in Trondheim, which have been a feeder university to search giants like Yahoo, Google and Microsoft.

With the tidal growth of data (particularly unstructured) in the past decade, it is no surprise that enterprise search has seen impressive increases in levels of demand. Hence, says Dr. Olstad, the competition for bright young knowledge management or search graduates has intensified. He also points out that several search start-ups have risen out of Norway, benefitting from the knowledge and experience in the local search ecosystem.

One example of these Norwegian search start-ups is Comperio, which has matured to being 50 employees strong with offices in Oslo, Stockholm, London and Boston. Comperio is a Systems Integrator (SI) focused on implementing FAST Search projects for enterprise customers. These search solutions, more recently developed on the FAST Search for SharePoint platform, offer an alternative to traditional integration between IT systems. ‘Today, a lot of IT budgets devoted to integrating old and new solutions. It costs time and money. We rather use search technology’, says Comperio founder and CEO Jørn Ellefsen.

Comperio has developed search solutions in Norway for such recognisable names as, Sintef, Posten, DSS, DNV and Innovation Norway. Outside of Norway, Comperio has delivered large projects for the likes of UBS and Shell. ‘The search-based technology we are developing is largely generic, and it allows us to reuse the solutions from previous projects with new customers. This means that it is both cheaper and better for everyone. This market is starting to wake up in earnest and will grow quickly when you discover how effective the technology is. I think the market for our services will increase tenfold over the next three years’, added Ellefsen.

Comperio are currently hiring for their Oslo and London offices. See the open positions @ http://www.comperiosearch.com/about-comperio/work-for-us/.

Original article from http://www.tu.no/it/article292738.ece.