Keynote

Fighting the crowds to find a seat for the keynote at Day 2 at elastic{ON}15 we were blocked by a USB stick with the curious caption  Microsoft (heart) Linux. Things have certainly changed.

Microsoft

The keynote, led by Elastic SVP of sales Aaron Katz, included Pablo Castro of Microsoft who was keen to explain how this probably isn’t so far from the truth. Elasticsearch is used  internally in several Microsoft products among Linux and other open source software and this is a huge change from the Microsoft we know from around five years ago. Pablo revealed some details towards how elasticsearch is used as data storage and search platform in MSN, Microsoft Dynamics and Azure Search. Microsoft truly has gone through some fundamental changes lately embracing open source both internally and externally. We see this as a demonstration of the power of open source and the huge value of Elastic(search) brings to  many organizations. As Jordan Sissel said in the keynote yesterday “If a user has a problem, it is a bug”. This is a philosophical stance towards a conception of software as an enabler of  creativity and growth, in contrast to viewing software as a fixed product packaged for sale.

Goldman Sachs

Microsofts contribution was in the middle part of the keynote. The first part was a discussion with Don Duet, managing director of Goldman Sachs. Goldman Sachs provides financial services on a global scale, and has been on the forefront of technology since its inception in 1869. They were an early adopter of Elasticsearch since it was as an easy to use search and analytics tool for big data. Goldman Sachs is now using elasticsearch extensively as a key part of their technological stack.

NASA

The most mind blowing part of the keynote was the last one held by two chaps from the Jet Propulsion Labs team at NASA, Ricky Ma and Don Isla. They first showed their awesome internal search with previews, and built in rank tuning. Then they talked about the Mars Curiosity rover, a robot planted on Mars which runs around taking samples and selfies. It constantly sends data back to earth where the JPL team analyzes the operations of the rover. Elasticsearch is naturally at the center of this interplanetary operation, nothing less.

Mars Curiosity Rover Selfie

The remainder of the day contained sessions across the same three tracks as the first day. In addition five tracks of birds of a feather or “lounge” sessions were held where people gathered in smaller groups to discuss various topics.  Needless to say the breadth of the program meant we were stretched thin. We chose to focus on three topics that are of particular importance to our customers: aggregations, security & Shield, and resiliency

More aggregations

Adrien Grand & Colin Goodheart-Smithe did a deep dive into the details of aggregations and how they are computed. In particular how to tune them and the results in terms of execution complexity. A key point is the approximations that are employed to compute some of the aggregations which involve certain trade offs in speed over accuracy. Aggregations are a very powerful feature requiring some some planning to be feasible and efficient.

Security/Shield

Uri Boness talked about Shield and the current state of authentication & authorization, He provided some pointers to what is on the roadmap for the coming releases. Unfortunately, there does not appear to be any concrete plans for providing built in document level security. This is a sought after feature that would certainly make the product more interesting in many enterprise settings. Then again, there are companies who provide connector frameworks that include security solutions for elasticsearch. We had a chat with some of them at the conference, including Enonic, SearchBlox and Search Technologies.

Facebook

Peter Vulgaris from Facebook explained how they are using elasticsearch. To me, the story resembled Microsoft’s. Facebook has heaps of data, and lots of use cases for it. Once they started to use elasticsearch it was widely adopted in the company and the amount of data indexed grew ever larger which forced them to think more closely about how they manage their clusters.

Resiliency

Elasticsearch is a distributed system, and as such shares the same potential issues as other distributed systems. Boaz Leskes & Igor Motov explained the measures that have been undertaken in order to avoid problems such as “split-brain” syndrome. This is when a cluster is confused about what node should be considered the master. Data safety and security are important features of Elasticsearch and there is a continuous effort in place in these areas.

Lucene

Stepping back to day 1 and the Lucene session featuring the mighty Robert Muir, we learned that Lucene version 5 includes a lot of improvements. Especially performance wise regarding compression both on indexing and query times which enables faster execution times and reduces resource consumption. There has also been made efforts to the Lucene core enabling a merging of query and filter as two sides of the same coin. After all a query is just  a filter with a relevance score. On another note Lucene will now handle caching of queries by itself.

Wrapping it up

Elastic{ON}15 stands as a confirmation of the attitude that were essential in the creation of the elasticsearch project. The visions that guided the early development are still valid today, except the scale is larger. The recent emphasis on stability, security and resiliency will welcome a new wave of users and developers.

At the same time there is a continuous exploration and development into big data related analytics but with the speed and agility we have come to expect from Elasticsearch.

Thanks for this year, looking forwards to next!

Noen utvalgte bilder fra arrangementet:

Download the presentations:

• Leveraging Search Based Applications on new Intranet
  (David Wright, Lead SharePoint Architect, Coutts)
• Searching Internally Using Enterprise Social Networks
  (Zane Freame, Technical Sales Professional, Microsoft UK)
• Enterprise Search Strategy by Comperio Search
  (Job Maelane, John Thompson, Comperio Search)

Some pictures from the event…

Comperio Search welcomes its guests...

Zane Freame, Technical Sales Professional, Microsoft UK

David Wright, Lead SharePoint Architect, Coutts

Trond Renshuslokken, Comperio CEO

The release of SharePoint 2013 is touted by Microsoft as a bridge from the past to the future; a pivotal shift from server- and wave-based releases to a rapid release cycle, combining the muscles from the cloud, fresh design principles and Enterprise Social technology. They’re proud, and they should be.

In the keynote, Microsoft also revealed some of the rationale behind the recent acquisition of Yammer, seeing how it complements SharePoint’s strong document-management skills with Yammer’s deep expertise in making the enterprise more social.

Jeff Teper, another high-profile SharePoint executive, told the crowds that over the last three years, Microsoft’s engineering team has centralized their effort around three important pillars: Experiences, Innovation and Ecosystem. This shows off in a dramatically improved user experience, better development tools and a completely revamped application model paving the way for us partner companies to more rapidly and with more flexibility deploy innovative applications in the cloud.

Another consistent theme throughout the keynote, was how Search now has a bigger role across SharePoint and throughout the Microsoft stack. FAST is now fully integrated, and the new search core drives both content and recommendations throughout the suite.

We now have a much better baseline when implementing Search technology in the enterprise. Hopefully this is finally the time when custom implementations of Enterprise Search will stop ending up as just a search box, and instead unleash the potential of Your organization’s accumulated knowledge and tackle today’s overwhelming growth of information.

FAST ESP was, and still is, an incredibly flexible and scalable enterprise search platform and is used by customers with sophisticated search needs and a significant requirement for updating the frequency, relevance adjustments, scalability and stability. The platform has very rich semantic and linguistic characteristics, and a wide range of connectors to the underlying systems, and connection points which can search and present relevant content adapted to the users context and device. These features generate revenue and/ or reduce costs for more than 2,600 customers worldwide that use or have used the FAST ESP platform.

FAST Search and Transfer was acquired by Microsoft in 2007. Following the acquisition, it became clear that FAST ESP will not be continuing as an independent cross-platform search engine. FAST ESP will however, exist in mainstream support until 2013 (2015 for customers with a License Grant), and extended support on all OS platforms until 2018 (2020 for License Grant customers).

During this period it is possible and safe to use and even develop new services on the FAST ESP search engine. Our team of Application Management Services (AMS) consultants have extensive experience in the FAST and FAST ESP platform – many of them formerly worked in consultancy, support or services in Fast Search and Transfer / Microsoft.

While Comperio’s AMS services can customise and sustain stable operation of customers on the FAST ESP platform, we also actively advise on search strategy, design, solution architecture and new search technology, as needs and technology evolve.

Our long-term plan with Comperio Front has been making customers less dependent on the underlying search engine, and the situation with FAST ESP shows that this has been a good strategy. For our customers with Comperio Front, we are now able to provide effective transitions to new technology platforms, as well as new corporate business models, and the search engine’s position as an information hub are preserved in Comperio Front.

When it comes to transitions to new technologies post the FAST ESP era, more choices and opportunities are opening up. For many traditional FAST ESP customers in e-commerce, catalog, classified ads and media, FAST for SharePoint is not necessarily a natural choice straight away, especially if they use anything other than Microsoft technology. Comperio has experience and dialogue with several customers on the transition to open source platforms and cloud services such as Solr, Elasticsearch, Amazon Cloud Search and also Google Custom Search – when called for.

That said, Microsoft plans to further use the advanced capabilities of FAST in their products, both in Office/SharePoint/Exchange/Office365, as well as in its search engine on the internet – Bing. Microsoft’s release of FAST Search for SharePoint 2010 is an example of this and the next generation of Microsoft products will show this more clearly. For many FAST ESP customers in the business market, it is natural to follow this path, as long as it is possible to build the solutions they want. And this is also where Comperio Front comes in, as a building block to make it possible to customize the search based solutions in SharePoint in a flexible manner.

Microsoft’s development team for search, or FAST R&D department, is based in Torggata, Oslo, and works closely with both the Bing and the Microsoft Office Division development teams.
With Comperio’s close affiliation from a geographical, historical and technical perspective, we continue to follow the FAST / Microsoft team in their development (and vice-versa) with technical and strategic dialogue constantly held.

But above anything else, we want to help our clients to great search experiences!

You can learn more about FAST ESP at http://www.comperiosearch.com/products/fast-esp/ and go to www.comperiosearch.com to see other products and services offered by Comperio.

FAST ESP var og er fremdeles en utrolig fleksibel og skalerbar enterprise search platform og blir brukt av kunder med avanserte søkebehov og ekstreme behov for oppdateringshyppighet, relevansjusteringer, skalerbarhet og stabilitet. Plattformen har meget rike semantiske og lingvistiske egenskaper, og et stort utvalg av connectorer til underliggende systemer, samt tilkoblingspunkter for å kunne søke og presentere relevant innhold tilpasset brukerens kontekst og device. Disse egenskapene er inntektskilde og kostnadsreduserende faktor for mer enn 2600 kunder verden over som benytter eller har benyttet FAST ESP plattformen.

FAST Search and Transfer ble kjøpt av Microsoft I 2007. Etter oppkjøpet er det etter hvert blitt klart at FAST ESP ikke blir videreført som selvstendig crossplattform søkemotor. FAST ESP vil imidlertid eksistere i mainstream support fram til 2013 (2015 for kunder med en Licence Grant), og extended support på alle OS plattformer fram til 2018 (2020 for License Grant kunder).

I denne perioden er det fullt mulig og trygt å benytte og til og med utvikle nye tjenester på FAST ESP søkemotor. Vårt team av Application Management Services  (AMS) konsulenter har lang fartstid fra FAST og med FAST ESP plattformen., og har blant annet jobbet med konsulenttjenster, AMS og support internt hos Fast Search and Transfer/Microsoft. Samtidig som vi med våre AMS tjenester kan oppretteholde stabil drift av din FAST ESP plattform, vil vi være aktive rådgivere og løsningsarktekter over på en ny søketeknologi ettersom behovene melder seg.

Vår langsiktige plan med Comperio Front har vært å gjøre våre kunder mindre avhengige av underliggende søkemotor, og situasjonen med FAST ESP viser at dette har vært en god anbefaling. For våre kunder med Comperio Front er vi nå i stand til å tilby effektive  overganger til nye teknologiplattformer, samtidig som bedriftens forretningslogikk og søkemotorens posisjon som  informasjonsnav er ivaretatt i Comperio Front.

Når det gjelder overgangene til nye teknologier i en etter FAST ESP- era er det flere valg og muligheter som åpner seg. For mange tradisjonelle FAST ESP kunder innen ehandel, katalog, rubrikk og media er ikke FAST for Sharepoint nødvendigvis er et naturlig valg for øyeblikket, spesielt ikke dersom man ellers benytter noe annet enn Microsoft teknologi. Comperio har erfaring og dialog med flere kunder om overgang til opensource plattformer og Cloudtjenester som Solr, Elasticsearch og Amazon Cloudsearch, og også Google Custom Search, alt etter behov.

Når dette er sagt er Microsoft’s plan framover å bruke de avanserte egenskapene til FAST i sine produkter videre, både for kontorstøtte i Office/Sharepoint/Exchange/Office365, så vel som i sin søkemotor på internett – Bing. Microsofts release av FAST Search for Sharepoint 2010 er et eksempel på dette, og neste generasjons Microsoft produkter vil vise dette enda tydeligere. For mange FAST ESP kunder i bedriftsmarkedet er det naturlig å følge dette sporet, så lenge det er mulig å bygge de løsningene man ønsker. Og også her er kommer Comperio Front inn som en byggestein for å gjøre det mulig å tilpasse søkebaserte løsninger i Sharepoint på en fleksibel måte.

Microsofts utviklingsteam for søk er lokalisert med basis i FAST R&D avdeling i Torggata Oslo, og samarbeider tett med så vel Bing som Microsoft Office sine utviklingsteam. Med vår nære tilknytning, geografisk, historisk og teknisk fortsetter Comperio å følge FAST/Microsoft i deres videre utvikling, og vårt Microsoft team er hele tiden i Front med det nye som leveres fra Microsoft.

Men først av alt ønsker vi å hjelpe våre kunder til gode søkeopplevelser!

Comperio consultant, Marcus Johansson, co-authors ‘Working with Microsoft FAST Search Server 2010 for SharePoint’.

As most search techies will testify, the life of a consultant working on complex enterprise search projects with large customers, can be quite a demanding one. To commit to investing personal time, outside of all this project work is, to put pen to paper and document a series of learnings and tips in a structured and user-friendly format, is no mean feat. So on this note, we salute Comperio consultant Marcus Johansson and congratulate him on the release of his first book earlier this month - Working with Microsoft FAST Search Server 2010 for SharePoint.

Marcus co-authored the book with Mikael Svenson and Robert Piddocke, who are also highly regarded contributors to the search community. Here, in Marcus’ own words, is some background on the book:

The book is split into two parts: the first one explains what you’ll need to know to deploy and administrate a solution. The second one is targeting developers who want to build their own search solutions on top of FAST Search for SharePoint. Although we expect most readers to have a SharePoint background, people who’s worked with FAST technology in previous incarnations will hopefully feel at home as well.

Recognizing  FAST Search for SharePoint’s popularity, and seeing that the product is many people’s first exposure of Enterprise Search technology, the first two chapters introduce the reader to important concepts and terminology. We felt it was critical to get this backdrop right as there are vastly different data and user experience opportunities (and challenges) in Search technology, than in e.g. a database-driven solution.

The next few chapters  target the IT Pro audience and deal with such things as architecture, scale-out, deployment and security. Scaling, in particular, is covered in depth seeing how FAST Search for SharePoint is often used for top-tier solutions both in  query load and content volume. Additionally, day-to-day operations and the various methods of maintenance is covered in detail. We show you how to interact with and follow-up the system through the SharePoint GUI, but also how to work with the solutions through PowerShell and, for those adventurous people who wants to truly integrate the platform in their application environments, how you can integrate against FAST Search for SharePoint’s native APIs even for administrational and operational tasks.

The second part of the book is a good fit for application developers who want to create their own search solutions on top of FAST Search for SharePoint, whether that means extending a default FAST Search Center or completely build their own search-driven application truly taking advantage of the powerful platform. Doing so, developers will realize an Enterprise Search platform, such as FAST Search for SharePoint, relies not only on a competent and flexible index, but also on a framework for advanced content processing and the possibilities to query into the data using a rich query language.

Finally, the book is wrapped up with a tutorial-like section on how to resolve common problems and how to attack frequent development scenarios.

At Comperio, we see how more and more of our clients realize the potential in not only implementing a global search experience in their enterprise, but to deploy targeted search-driven applications across their organizations. One user group might need a mash-up of data aggregated from several internal content sources, and perhaps another group needs to trigger a certain function that he or she perceives as a single operation, but in reality hits 2-3 different source systems in the backend. A search-driven interface is the very sweet spot for building such applications, and  I’m hopeful this book will give you the tools of the trade to implement them in reality.

Happy reading!

In recognition of Marcus’ achievement, the team here in Comperio have given him temporary clearance to use lines like ‘As I say in my book, Working with Microsoft FAST Search Server 2010 for SharePoint, …’ in internal meetings and possibly even at customer presentations.

The book is now available for purchase on Amazon - or you can win one of three free copies being offered by Comperio, by being one of the first to tweet this post mentioning @comperiosearch.

Dr. Olstad speaks on how Norway is a hotbed for search technology development and this search ecosystem has it’s source at NTNU in Trondheim, which have been a feeder university to search giants like Yahoo, Google and Microsoft.

With the tidal growth of data (particularly unstructured) in the past decade, it is no surprise that enterprise search has seen impressive increases in levels of demand. Hence, says Dr. Olstad, the competition for bright young knowledge management or search graduates has intensified. He also points out that several search start-ups have risen out of Norway, benefitting from the knowledge and experience in the local search ecosystem.

One example of these Norwegian search start-ups is Comperio, which has matured to being 50 employees strong with offices in Oslo, Stockholm, London and Boston. Comperio is a Systems Integrator (SI) focused on implementing FAST Search projects for enterprise customers. These search solutions, more recently developed on the FAST Search for SharePoint platform, offer an alternative to traditional integration between IT systems. ‘Today, a lot of IT budgets devoted to integrating old and new solutions. It costs time and money. We rather use search technology’, says Comperio founder and CEO Jørn Ellefsen.

Comperio has developed search solutions in Norway for such recognisable names as, Sintef, Posten, DSS, DNV and Innovation Norway. Outside of Norway, Comperio has delivered large projects for the likes of UBS and Shell. ‘The search-based technology we are developing is largely generic, and it allows us to reuse the solutions from previous projects with new customers. This means that it is both cheaper and better for everyone. This market is starting to wake up in earnest and will grow quickly when you discover how effective the technology is. I think the market for our services will increase tenfold over the next three years’, added Ellefsen.

Comperio are currently hiring for their Oslo and London offices. See the open positions @ http://www.comperiosearch.com/about-comperio/work-for-us/.

Original article from http://www.tu.no/it/article292738.ece.