I would argue that 33 lines for creating the facets above is too much.

To save you from the dreaded hassle of writing JSON in your programs, I created addict, a Python dictionary that let’s you create nested dictionaries using get- and set attribute syntax.

Here’s how the same code looks with addict:

addict is open-source and MIT-licensed. It is actively developed and anyone interested is encouraged to submit issues and pull requests, as both are more than welcome.

• This has been posted earlier on lab.klemespen.com.
• Even though some of these ideas are not what you’d normally implement in a business environment, some of the concepts can obviously be transferred over to businesses trying to provide an efficient workplace for its employees.
• Norch is developed by Fergus McDowall, an employee of Comerio.

What if you could index your whole life and make this lifeindex available through search? What would that look like, and how could it help you? Refinding information is obviously one of the use case for this type of search. I’m guessing there’s a lot more, and I’m curious to figure them out.

Actions and reactions instead of web pages

I had the lifeindex idea for a little while now. Originally the idea was to index everything I browsed. From what I know and where Norch is, it would take a while before I was anywhere close to achieving that goal. Then I thought of IFTTT, and saw it as a ‘next best thing’. But then it hit me that now I’m indexing actions, and that’s way better than pages. But what I’m missing from most sources now are the reactions to my actions. If I have a question, I also want to crawl and index the answer. If I have a statement, I want to get the critique indexed.

IFTTT and similar services (like Zapier) is quite limiting in their choice of triggers. Not sure if this is because of choices done by those services or limitations from the sites they crawl/pull information from.

A quick fix for this, and a generally good idea for Search Engines, would be to switch from a preview of your content to the actual content in the form of an embed-view. Here exemplified:

Will embed-view of your content replace the preview-pane in modern #search #engine solutions? Why preview when you can have the real deal?

— Espen Klem (@eklem) November 24, 2014

Technology: Hello IFTTT, Google SpreadSheet and Norch

IFTTT is triggered by my actions, and stores some data to a series of spreadsheets on Google Drive. These spreadsheets can deliver JSON. After a little document processing these JSON-files can be fed to the Norch-indexer.

Why hasn’t this idea popped up earlier?

Search engines used to be hardware guzzling technology. With Norch, the “NOde seaRCH” engine, that has changed. Elasticsearch and Solr are easy and small compared to i.e. SharePoint Search, but still it needs a lot of hardware. Norch can run on a Raspberry Pi, and soon it will be able to run in your browser. Maybe data sets closer to small data is more interesting than big data?

Norch running on a Raspberry Pi

Why using a search engine?

It’s cheap and quick. I’m not a developer, and I’ll still be able to glue all these sources together. Search engines are often a good choice when you have multiple sources. IFTTT and Google SpreadSheet makes it even easier, normalising the input and delivering it as JSON.

How far in the process have I come?

So far, I’ve set up a lot of triggers/sources at IFTTT.com:

• Instagram: When posting or liking both photos and videos.
• Flickr: When posting an image, creating a set or linking a photo.
• Google Calendar: When adding something to one of my calendars.
• Facebook: When i post a link, is tagged, post a status message.
• Twitter: When I tweet, retweet, reply or if somebody mentions me.
• Youtube: When I post or like a video.
• GitHub: When I create an issue, gets assigned to an issue or any issues that I part take in is closed.
• WordPress: When new posts or comments on posts.
• Android location tracking: When I enter and exit certain areas.
• Android phone log: Placed, received and missed calls.
• Gmail: Starred emails.

And gotten a good chunk of data. Indexing my SMS’es felt a bit creepy, so I stopped doing that. And storing email just sounded too excessive, but I think starred emails would suit the purpose of the project.

Those Google Drive documents are giving me JSON. Not JSON that I can feed directly Norch-indexer, it needs a little trimming.

Issues discovered so far

Manual work

This search solution needs a lot of manual setup. Every trigger needs to be set up manually. Everytime a new trigger is triggered, I get a new spreadsheet that needs a title row added. Or else, the JSON variables will look funny, since first row is used for variable names.

The spreadsheets only accepts 2000 rows. After that a new file is created. Either I need to delete content, rename the file or reconfigure some stuff.

Level of maturity

IFTTT is a really nice service, and they treat their users well. But, for now, it’s not something you can trust fully.

Cleaning up duplicates and obsolete stuff

I have no way of removing stuff from the index automatically at this point. If I delete something I’ve added/written/created, it will not be reflected in the index.

Missing sources

Books I buy, music I listen to, movies and TV-series I watch. Or Amazon, Spotify, Netflix and HBO. Apart from that, there are no Norwegian services available through IFTTT.

History

The crawling is triggered by my actions. That leaves me without history. So, i.e. new contacts on LinkedIn is meaningless when I don’t get to index the existing ones.

Next steps

JSON clean-up

I need to make a document processing step. Norch-document-processor would be nice if it had handled JSON in addition to HTML. Not yet, but maybe in the future? Anyway, there’s just a small amount of JSON clean-up before I got my data in and index.

When this step is done, a first version can be demoed.

UX and front-end code

To show the full potential, I need some interaction design of the idea. For now they’re all in my head. And these sketches needs to be converted to HTML, CSS and Angular view.

Embed codes

Figure out how to embed Instagram, Flickr, Facebook and LinkedIn-posts, Google Maps, federated phonebook search etc.

OAUTH configuration

Set up OAUTH NPM package to access non-public spreadsheets on Google Drive. Then I can add some of the less open information I have stored.

Here is a quick demo that demonstrates how to create an “instant search” service using JSON, ajax and JQuery. This instant search application uses the excellent OpenSearch interface provided by Wikipedia. Use this code as a starting point for developing prettier result pages.

The Wikipedia query API is described in some detail here.

Click here to see the demo, or copy and paste the code below into a blank HTML document.

<script type="text/javascript" src="http://code.jquery.com/jquery-latest.js"></script>

Search Wikipedia!

<input id="searchterm" type="text" /> <button id="search">search</button>

<script type="text/javascript">// <![CDATA[
      $("#searchterm").keyup(function(e){
        var q = $("#searchterm").val();
        $.getJSON("http://en.wikipedia.org/w/api.php?callback=?",
        {
          srsearch: q,
          action: "query",
          list: "search",
          format: "json"
        },
        function(data) {
          $("#results").empty();
          $("#results").append("

Results for <b>" + q + "</b>

");
          $.each(data.query.search, function(i,item){
            $("#results").append("
<div><a href='http://en.wikipedia.org/wiki/" + encodeURIComponent(item.title) + "'>" + item.title + "</a>" + item.snippet + "</div>
");
          });
        });
      });

// ]]></script>

This image search app provides a neat solution to the history/speed issue by simply displaying results in a push-pop stack. As you type in new search terms, the image thumbnails are prepended to the resultset. Your search history stays on the page, but is simply pushed further down as new thumbnails appear. Its is really fast and simple to browse your image search history.

Also, this search app operates entirely in 30 lines of HTML and JavaScript by using a JSON feed to the flickr API. Simple and elegant.

Click here for the demo

Or copy and paste the code below this line:

<html>
  <head>
    <script src="http://code.jquery.com/jquery-latest.js"></script>
    <title>Comperio Super Simple Image Search</title>
  </head>
  <body>
    Search for <b>cats, dogs, cakes</b>, or anything else that takes your fancy :-)
    <br />
    <input id="searchterm" />
    <button id="search">search</button>
    <div id="results"></div>
    <script>
      $("#search").click(function(){
        $.getJSON("http://api.flickr.com/services/feeds/photos_public.gne?jsoncallback=?",
        {
          tags: $("#searchterm").val(),
          tagmode: "any",
          format: "json"
        },
        function(data) {
          $.each(data.items, function(i,item){
            $("<img/>").attr("src", item.media.m).prependTo("#results");
            if ( i == 10 ) return false;
          });
        });
      });
    </script>
  </body>
</html>

Update: Aug 5th 2012 A live demo of asynchronous search results with JQuery, Solr, Json and Ajax can now be found here

There are many reasons to load search results asynchronously- most of the big searchy dotcoms are going in this direction with Google, Facebook and Twitter all relying heavily on ajax to render their feed-based pages.

Considering how mainstream it has become, information on the nuts and bolts of the technique remain thin on the ground. So here is a really quick overview of what you need to get started.

For the purposes of this post I will assume that you have an working installation of Solr in a more or less standard configuration. Getting Solr up and running is no mean feat in itself, but mercifully there are a number of fairly decent guides squirreled away on the internet. You could of course use any search motor that returns search results in a Json format, in fact even if they dont, Comperio Front can now be used to provide a JSON query service on any (collection of) search motor(s).

So lets assume that a query for “Ford” returns Json in the following format from our very small index of cars. Note that the Solr URL for this feed would look something like this: http://localhost:8080/solr/select/?q=ford&wt=json&json.wrf=?&indent=true where wt specifies json and json.wrf specifies the callback

({
  "responseHeader":{
    "status":0,
    "QTime":1,
    "params":{
      "json.wrf":"?",
      "indent":"true",
      "q":"ford",
      "wt":"json"}},
  "response":{"numFound":3,"start":0,"docs":[
      {
        "id":"147",
        "name":"Ford Capri",
        "description":"pink 1978 model- great condition",
        "owner":"576463865",
        "store":"59.7440738,10.204456400000026",
        "tags":[
          "Ford, Capri, Pink, 1978"]},
      {
        "id":"148",
        "name":"Ford Orion",
        "description":"Blue, Good runner",
        "owner":"576463865",
        "store":"59.9138688,10.752245399999993",
        "tags":[
          "blue, orion, 1.6"]},
      {
        "id":"149",
        "name":"Ford Anglia",
        "description":"As seen in Harry Potter, Can be flown",
        "owner":"576463865",
        "store":"57.2366283,-2.2650615999999673",
        "tags":[
          "Harry Potter, Anglia, Flying car"]}
    ]
  }
})

We can now parse this feed by using the .getJSON function in the excellent jQuery library. The javascript and associated HTML would be more or less like so:

<div id="rs"></div>
  <script type='text/javascript' src='//ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.js'></script>
  <script type='text/javascript'>
    $.getJSON("http://localhost:8080/solr/select/?q=ford&wt=json&json.wrf=?&indent=true", function(result){
    var Parent = document.getElementById('rs');
    for (var i = 0; i < result.response.docs.length; i++) {
      var thisResult = "<b>" + result.response.docs[i].name + "</b><br>" + result.response.docs[i].description
      + ", " + result.response.docs[i].tags + "<br>";
      var NewDiv = document.createElement("DIV");
      NewDiv.innerHTML = thisResult;
      Parent.appendChild(NewDiv);
    }
  });
</script>

And thats it :-) You will notice that when the HTML above is loaded in a browser, that the search results are returned independently of the webpage.

Next Steps: Use JavaScript templating to render results from Solr