• This has been posted earlier on lab.klemespen.com.
• Even though some of these ideas are not what you’d normally implement in a business environment, some of the concepts can obviously be transferred over to businesses trying to provide an efficient workplace for its employees.
• Norch is developed by Fergus McDowall, an employee of Comerio.

What if you could index your whole life and make this lifeindex available through search? What would that look like, and how could it help you? Refinding information is obviously one of the use case for this type of search. I’m guessing there’s a lot more, and I’m curious to figure them out.

Actions and reactions instead of web pages

I had the lifeindex idea for a little while now. Originally the idea was to index everything I browsed. From what I know and where Norch is, it would take a while before I was anywhere close to achieving that goal. Then I thought of IFTTT, and saw it as a ‘next best thing’. But then it hit me that now I’m indexing actions, and that’s way better than pages. But what I’m missing from most sources now are the reactions to my actions. If I have a question, I also want to crawl and index the answer. If I have a statement, I want to get the critique indexed.

IFTTT and similar services (like Zapier) is quite limiting in their choice of triggers. Not sure if this is because of choices done by those services or limitations from the sites they crawl/pull information from.

A quick fix for this, and a generally good idea for Search Engines, would be to switch from a preview of your content to the actual content in the form of an embed-view. Here exemplified:

Will embed-view of your content replace the preview-pane in modern #search #engine solutions? Why preview when you can have the real deal?

— Espen Klem (@eklem) November 24, 2014

Technology: Hello IFTTT, Google SpreadSheet and Norch

IFTTT is triggered by my actions, and stores some data to a series of spreadsheets on Google Drive. These spreadsheets can deliver JSON. After a little document processing these JSON-files can be fed to the Norch-indexer.

Why hasn’t this idea popped up earlier?

Search engines used to be hardware guzzling technology. With Norch, the “NOde seaRCH” engine, that has changed. Elasticsearch and Solr are easy and small compared to i.e. SharePoint Search, but still it needs a lot of hardware. Norch can run on a Raspberry Pi, and soon it will be able to run in your browser. Maybe data sets closer to small data is more interesting than big data?

Norch running on a Raspberry Pi

Why using a search engine?

It’s cheap and quick. I’m not a developer, and I’ll still be able to glue all these sources together. Search engines are often a good choice when you have multiple sources. IFTTT and Google SpreadSheet makes it even easier, normalising the input and delivering it as JSON.

How far in the process have I come?

So far, I’ve set up a lot of triggers/sources at IFTTT.com:

• Instagram: When posting or liking both photos and videos.
• Flickr: When posting an image, creating a set or linking a photo.
• Google Calendar: When adding something to one of my calendars.
• Facebook: When i post a link, is tagged, post a status message.
• Twitter: When I tweet, retweet, reply or if somebody mentions me.
• Youtube: When I post or like a video.
• GitHub: When I create an issue, gets assigned to an issue or any issues that I part take in is closed.
• WordPress: When new posts or comments on posts.
• Android location tracking: When I enter and exit certain areas.
• Android phone log: Placed, received and missed calls.
• Gmail: Starred emails.

And gotten a good chunk of data. Indexing my SMS’es felt a bit creepy, so I stopped doing that. And storing email just sounded too excessive, but I think starred emails would suit the purpose of the project.

Those Google Drive documents are giving me JSON. Not JSON that I can feed directly Norch-indexer, it needs a little trimming.

Issues discovered so far

Manual work

This search solution needs a lot of manual setup. Every trigger needs to be set up manually. Everytime a new trigger is triggered, I get a new spreadsheet that needs a title row added. Or else, the JSON variables will look funny, since first row is used for variable names.

The spreadsheets only accepts 2000 rows. After that a new file is created. Either I need to delete content, rename the file or reconfigure some stuff.

Level of maturity

IFTTT is a really nice service, and they treat their users well. But, for now, it’s not something you can trust fully.

Cleaning up duplicates and obsolete stuff

I have no way of removing stuff from the index automatically at this point. If I delete something I’ve added/written/created, it will not be reflected in the index.

Missing sources

Books I buy, music I listen to, movies and TV-series I watch. Or Amazon, Spotify, Netflix and HBO. Apart from that, there are no Norwegian services available through IFTTT.

History

The crawling is triggered by my actions. That leaves me without history. So, i.e. new contacts on LinkedIn is meaningless when I don’t get to index the existing ones.

Next steps

JSON clean-up

I need to make a document processing step. Norch-document-processor would be nice if it had handled JSON in addition to HTML. Not yet, but maybe in the future? Anyway, there’s just a small amount of JSON clean-up before I got my data in and index.

When this step is done, a first version can be demoed.

UX and front-end code

To show the full potential, I need some interaction design of the idea. For now they’re all in my head. And these sketches needs to be converted to HTML, CSS and Angular view.

Embed codes

Figure out how to embed Instagram, Flickr, Facebook and LinkedIn-posts, Google Maps, federated phonebook search etc.

OAUTH configuration

Set up OAUTH NPM package to access non-public spreadsheets on Google Drive. Then I can add some of the less open information I have stored.

While both platforms are based on Lucene, Elasticsearch has been built for scaling from day one, while it is often said Solr added this as an afterthought. Developers also generally find it very easy to interact with since it uses a JSON-based API model. Much could be said about the differences between the two platforms, but without going into a long list of technical details, Elasticsearch felt like a breath of fresh air when we first laid eyes on it a few years ago.

Elasticsearch or Solr? Google trends will give you a nice input to the dilemma.

Today the company behind the product, announced they have raised a whopping 70M USD in funding, bringing the total up to over 100M USD in the last 18 months!

For everyone loving the product, and everyone considering what platform to go with for their enterprise search needs, this is excellent news! Even though it is an open source product, and everyone in theory can submit suggested changes, bugfixes etc – there’s definite value in having a group of full time, dedicated developers building the product.

This means the product and the ecosystem around it will continue to evolve, improve and increase for a long time with considerable force.

So – for all those wondering what horse to put their bets on for future open source development;  Picking the fastest running horse is often better than picking the horse that’s been around the longest.

Comperio is an official partner with Elasticsearch.

Please join us at our breakfast seminar in Oslo in a few days (June 11). Elasticsearch will be there to present alongside Microsoft and Google.

1. Learn more about search, both tech and UX.
2. Show that a nice search user experience doesn’t need a search input box.
3. Show a search that is light weight.
4. Generally, build something nice, quickly, to show off.

“But what is it?” you ask again. The second answer is that it’s a search application for recipes containing the most in-season vegetables, fruits, mushrooms, fish, shellfish, poultry and meat. So, a search without a search box giving you the recipes that will make you pick the freshest, most tasty food available at any given time of the year. In Norway, you can get strawberries the whole year around, but it’s only a month or two during the summer that they taste really good. and are cheap. This goes for a lot of different foods.

We’ll blog every step of the journey. First small hackathon is this Friday, January 7th. To test our ideas we’ve already crawled food recipes from Klikk.no, and transformed them into JSON-format by using Forage Document Processor from Fergus McDowell’s excellent Forage Search Engine.

Next step is to get an Amazon EC2-server up and running, install Elasticsearch, and feed it recipes in JSON-format. We’ll also make some rough mockups of how the user interface is going to be, and maybe, hopefully get a simple version of the Seasonal Food Recipe Web Application up and running. If we after a while get enough traffic, we’ll try to do some machine learning. And we’ll definitely test out different ways of doing relevancy tuning.

Sounds nice? This is work in progress, so check back every now and then for new blog posts.

Comperio consultant, Marcus Johansson, co-authors ‘Working with Microsoft FAST Search Server 2010 for SharePoint’.

As most search techies will testify, the life of a consultant working on complex enterprise search projects with large customers, can be quite a demanding one. To commit to investing personal time, outside of all this project work is, to put pen to paper and document a series of learnings and tips in a structured and user-friendly format, is no mean feat. So on this note, we salute Comperio consultant Marcus Johansson and congratulate him on the release of his first book earlier this month - Working with Microsoft FAST Search Server 2010 for SharePoint.

Marcus co-authored the book with Mikael Svenson and Robert Piddocke, who are also highly regarded contributors to the search community. Here, in Marcus’ own words, is some background on the book:

The book is split into two parts: the first one explains what you’ll need to know to deploy and administrate a solution. The second one is targeting developers who want to build their own search solutions on top of FAST Search for SharePoint. Although we expect most readers to have a SharePoint background, people who’s worked with FAST technology in previous incarnations will hopefully feel at home as well.

Recognizing  FAST Search for SharePoint’s popularity, and seeing that the product is many people’s first exposure of Enterprise Search technology, the first two chapters introduce the reader to important concepts and terminology. We felt it was critical to get this backdrop right as there are vastly different data and user experience opportunities (and challenges) in Search technology, than in e.g. a database-driven solution.

The next few chapters  target the IT Pro audience and deal with such things as architecture, scale-out, deployment and security. Scaling, in particular, is covered in depth seeing how FAST Search for SharePoint is often used for top-tier solutions both in  query load and content volume. Additionally, day-to-day operations and the various methods of maintenance is covered in detail. We show you how to interact with and follow-up the system through the SharePoint GUI, but also how to work with the solutions through PowerShell and, for those adventurous people who wants to truly integrate the platform in their application environments, how you can integrate against FAST Search for SharePoint’s native APIs even for administrational and operational tasks.

The second part of the book is a good fit for application developers who want to create their own search solutions on top of FAST Search for SharePoint, whether that means extending a default FAST Search Center or completely build their own search-driven application truly taking advantage of the powerful platform. Doing so, developers will realize an Enterprise Search platform, such as FAST Search for SharePoint, relies not only on a competent and flexible index, but also on a framework for advanced content processing and the possibilities to query into the data using a rich query language.

Finally, the book is wrapped up with a tutorial-like section on how to resolve common problems and how to attack frequent development scenarios.

At Comperio, we see how more and more of our clients realize the potential in not only implementing a global search experience in their enterprise, but to deploy targeted search-driven applications across their organizations. One user group might need a mash-up of data aggregated from several internal content sources, and perhaps another group needs to trigger a certain function that he or she perceives as a single operation, but in reality hits 2-3 different source systems in the backend. A search-driven interface is the very sweet spot for building such applications, and  I’m hopeful this book will give you the tools of the trade to implement them in reality.

Happy reading!

In recognition of Marcus’ achievement, the team here in Comperio have given him temporary clearance to use lines like ‘As I say in my book, Working with Microsoft FAST Search Server 2010 for SharePoint, …’ in internal meetings and possibly even at customer presentations.

The book is now available for purchase on Amazon - or you can win one of three free copies being offered by Comperio, by being one of the first to tweet this post mentioning @comperiosearch.