Add a small line of JavaScript in your template’s document ready function:

$("#MySearchBox").focus();

This will do two things for the user:

1. It’ll be easier to see the search box .
2. The user can start typing without having to click inside the search box.

Next issue is that most intranet and websites are more than just a search solution. Maybe you don’t want that much attention on the search box on your homepage. The solution is then to do this on your search result page.

This will make it easier for your users to enhance their search query when they’re not happy with the search result at hand.

Do you have any other examples on other quick fixes that could make an even better user experience for your search solution?

Last week, I saw the future. Wohaa, that’s always a great feeling.  I’ve seen it in earlier weeks also, but now it was even brighter than before. For me, it’s still called the Lebron Stack as Max Ogden explains it and consists of LevelDB, Browserify and npm. All this is mostly happening in JavaScript. Before I’m knocked to the ground: I wasn’t the first to either make the prediction or say it out loud. I’m way behind, and it’s not a very novel or extreme idea, just a really good one. But when something is predicted, it may take a long time before it happens, if it happens at all. I think it’s happening now-ish.

So this blog post is about why I think that time is now. Disclaimer for the .Net and Java heads And all you .Net- and Java-heads will surely find some stuff that will be done better within your part of the world, but hear me out! I know the list of “This already exists in OS [W] or [X]” or “You can do that with software [X], [Y] or [Z]“. I have these thoughts my self, and I’ve been wondering why I still think that Lebron and JavaScript still will be so much more important. I’m not saying that .Net and Java stuff will go away, it will just be less important (it already is) and most of the cool and stuff closer to the user will happen in the JavaScript world.

The future is bright at the Webrebels conference in Oslo, May – 2014.

Here are the reasons I found so far

• Most stuff happens in the browser Selling anything, you want to be where the people are. For regular people that’s on their smartphone using a web app or just a native app, which in most cases is a web app wrapped as a native app. Emerging markets makes this shift towards the browser happen even faster. The Firefox OS may fail as an OS, but still succeed creating a standard smartphone API for web applications, the WebAPI. This will make it even easier to create web apps for all of the world’s smartphones, which leads me on to my next point.
• Easier for startups and developers Competing with the big ones is never easy. Amazon Web Services, AWS,  and similar services made it a little easier to scale hardware use dynamically, and from that, the cost of hardware. With the browser as a VM and single page applications a lot of the web application rendering and logic is moved from the servers to the clients. So for a small company the choice is easy. Why do all the heavy lifting on your own servers when the users can do most of the application rendering and logic on their smartphones? The irony in the old “thin vs. thick client” debate is that the clients actually got a lot thinner, and in the same go started doing more of the heavy lifting. While a Google data center is impressive, I also got a feeling it’s a sign of something gone terribly wrong.
• Collaboration, modularity and minimum effort npm is great stuff. It takes away a lot of dependency pain in the JavaScript world. Combined with people that are very good at writing small modular programs and lots of stuff under the MIT license we have a winner. We now have tools for collaboration that actually works. People build their killer apps with very little effort on top of others’ greatness. No more reinventing the text editor.
• Cheaper hardware for regular users Okay, most people access the Internet through their phone, but the Chromebook explains this point very well. Why have a full OS, with all the hardware costs to run it fairly fast, when all you do is fire up a browser? The browser is the OS more and more each day. Last time my desktop at home broke down, I bought a new one. The new one was state of the art and it was a miscalculation buying it. Almost every time I boot it (running Ubuntu), I’m asked to upgrade to the newest version. That means every half year or so. The laptop I have, I actually use a little, but much less than my pad/tablet and phone.
• Everything fun is online Not a real argument, but hey… Isn’t it true?

But what about the Forage search engine you say?

So, what does these reasons for Lebron/JavaScript’s future success have to do with the Forage search engine? First of all, it’s written in JavaScript and needs very little hardware to run properly. You install it with npm, and that takes care of all the dependencies, like LevelDB, where the data is actually stored. Hopefully it will run in the browser in near future using Browserify and make testing, installing and maintaining search software so much easier and more accessible. It also opens up a lot of new interesting use cases for search. My guess is that it won’t compete with the bigger search engines, but that it will open up the possibility for better and cheaper search functionality for small scale solutions.

Anything you want to add to or subtract from the list?

Forage could then be added with a bookmarklet to any page (A bookmark adding a javascript to the page you’re on). With some simple UI-stuff you could define the Forage Document Processor Adapter, set up rules for Forage Crawler, crawl, process, index and then search within your indexed documents. All without using any servers, on premises or in the cloud. When the user is satisfied she or he could download the JSON-file with processed documents + scripts for adding a search box,  search result and navigators to a page.

Possible use cases for search server running inside browser:

• Easy site search setup
  One real benefit, and the initial idea, would be that the user would not need any server to test Forage and actually crawl a site. When page crawled the user can download JSON ready to be indexed + setup-files for a search box, navigators and search result. Or add it to a cloud service and there continue the work you started in your browser.
• A easy and modern search solution behind the firewall
  Behind the firewall, almost all software looks a bit duller, more beige and basically not modern. But through the browser you could easily combine the strength of  Forage and all the hidden gems behind a firewall. There would be some big issues with security, but for intranet and people search it could be a great solution.
• Ad hoc search on a site that is not yours
  Say you’re looking for something on a site. How about ad hock index it and then search it. Yes, it’s a bandwidth abuse waiting to happen, but could make a good tool for a lot of situations.
• Your life, searchable
  This may need a browser add on, but then again, maybe not. Anyway: How about your whole online life, searchable. Today you have your browser history. It shows you page title and page link. What if all the text and images was searchable?

Some UX sketches of the idea:

The user finds a page to crawl …

… clicks the bookmarklet …

... that adds Forage JavaScript-stuff to the page …

… much like a browser plugin or add on …

… tests a jQuery selector statement …

… and adds the field to the item when satisfied. Repeated until a full item is defined.

Here’s the feature suggestion at the Forage GitHub page. Ideas or comments are more than welcome! Want to know more about Forage? Check out the Forage GitHub-pages or stuff we’ve written about Forage.

EDIT: Drawn some new mock ups on the crawler part: Forage Fetch and written about the killer combo Lebron and what it will mean for search.

Search templates is a brand new feature  added to elasticsearch version 1.1. 0. It allows potentially complex search logic to be stored as templates. The user sends a query with the parameters that matters for him, along with the name of a search template, and the supplied parameters will be inserted at the proper place on execution.  The template format is moustache, a widely used javascript templating tool.

Cool stuff, you say. How can I use it? And does it really work?

In the current implentation of search templates, you create a JSON request specifying both the template and the parameters.

You can save the templates in the config/scripts folder of your elasticsearch installation, and use the saved query.
The template is saved as config/scripts/ingredientsquery.moustache. This is how my template looked like:

And here is a sample query using this template

To make changes to the template effective, you need to restart elasticsearch.

You would perhaps expect it to be possible to upload a search template using the REST API.  Unfortunately,  that hasn’t been implemented yet.  At the moment, you have to store the search templates in the  config/scripts folder in your elasticsearch node.  There is an issue registered to add this feature, but in the mean time, our friends over at qbox are helpful as always and offer to manually place the templates where they belong while we wait for elasticsearch version 1.1.1 (or something).

Using this awesome feature, the only reason we still need some middleware between our elasticsearch installation and our browser is to protect the search service from malicious requests. As it so happens, Qbox have recently added a feature that enables public read only access to the elasticsearch query _search endpoint.

This enables us to use the Qbox elasticsearch service directly from the angularjs client for elasticsearch.

Thus we eliminate the need for the searchclient in ruby, and one less server component to care about.

Sounds nice? This is work in progress, so check back every now and then for new blog posts.

0.3.0 fixes lots niggles with indexing, and gives a pretty hefty improvement to memory usage. There is also a built in matcher for creating cool auto-suggest and auto-complete controls based on the content of the corpus.

In related news there is now a family of crawling tools which allow you to scrape, process and index web content into your Forage server. Check out forage-fetch, forage-document-processor and forage-indexer on the forage.js GitHub group.

As always, feedback, pull requests, comments, praise, criticism and beer are most welcome.

In this blog post we try to explain how to build a single-page application (SPA) to search in elasticsearch using AngularJS. For simplicity, we will use a twitter index which you can easily create yourself (see here and here).
Angular is an open-source JavaScript framework maintained by Google. According to the official Angular website “Angular teaches the browser new syntax through a construct we call directives”—in other words, Angluar allows you to extend the HTML syntax to make your application ‘more dynamic’. With Angular, we can also build applications following the Model-View-Control (MVC) design pattern.

To communicate with elasticsearch, we rely on elastic.js, a JavaScript client for elasticsearch.

In this post, we assume some basic understanding of search concepts and Angular. If you are not familiar with Angular, see the further readings section.

Sketching the structure out

Angular applications, in general, consist of three types of components: models, views and controllers, and our application is no exception.
The model in our application is, more or less, included within elastic.js, which sends requests to elasticsearch and returns the requested information as JSON objects. The views are almost plain HTML pages which allow the user to enter search queries and display the results of those queries. The controllers are mediators between elastic.js and the views. We have two views and two corresponding controllers for Twitter and Stackoverflow, but we will not be implementing the Stackoverflow part; we added it only for the sake of clarity.

The full code is on github, you can start by looking at the general structure of the application and then come back here to understand how the different parts of this application work together.

The single page in the single-page application

Single-page applications contain, as the name suggests, one page whose content changes dynamically. This page is called index.html in our search application. The following code snippet shows the main parts of index.html.

There are two important points to note about the code above. First, the Angluar directive np-app within the <html> tag which defines the boundary of the Angular application. Second, the ng-view directive which specifies where the different views will be rendered within the layout of index.html. Said differently, ng-view is ‘replaced’ by the dynamic content in the main page.

searchApp.js

In each and every Angular application, there is at least one Angular module. “searchApp” is the sole and main module in our application, as indicated in ng-app above. The following piece of code defines this module with its dependencies.

The module searchApp depends on three other modules, namely: (1) elasticjs.service, the JavaScript client for elasticsearch, elastic.js, (2) ngSanitize, an Angular module to sanitize HTML, and (3) infinite-scroll, a directive to implement infinite scrolling in Angular.

We also need to configure the so-called routes, URLs linking to views and controllers. In concrete terms, we need to link the items in the navigation bar to HTML partials. To do so, we use the route service, $routeProvider. Observe that each route has a view, templateUrl, and a controller. Recall that we used the directive ng-view to define the dynamic part within the main page, $routeProvider and ng-view are used together to achieve this dynamic behavior.

Twitter view

The first thing we need in a search application is a search box, ha? The following code snippet defines an instant search box that calls a search function, search(), every time the input text is changed. This search() function will be defined later in the Twitter controller.

Observe that we bind the input text to a property named queryTerm using the Angular directive ng-model. This binding makes the input value available throughout the Angular application via an object called scope ($scope). It is noteworthy that binding in Angular is two-way, meaning that if the input text (queryTerm) is changed somewhere in the application this change will be directly reflected on the user interface and vice versa.
Now let’s dive into how the search function works, and then we can come back to displaying the search results.

Simple queries

All the search functionalities are defined within the Twitter controller, TwitterController.js. We start by implementing a simple search function and then gradually enhance it. First, we define a controller named Twitter, which depends on $scope and esjResource (the elastic.js client).

Within the twitter controller, we instantiate the elastic.js client, passing the URL of the local elasticsearch instance. All search requests to elasticsearch go through a Request object, statusRequest, which queries the specified index and type.
Observe that we are using the binding $scope.queryTerm to access the search query. We use the match query family in elasticsearch to search in the _all field for $scope.queryTerm. In addition, we define the fields to be returned by elasticsearch instead of returning full documents. Finally, the search results are appended to $scope.resultArr (again defined within the $scope to make accessible by the views).

With the code above, however, we only get the top ten results matching the user’s query, usually we want to show more than that, this is where pagination comes into play.

Pagination with infinite scroll

Each search request by elastic.js returns ten results by default, but the user’s information need might not be satisfied by the top 10 results, hence we want to allow the user to load more results if needed. The process of dividing the search results into different pages is called pagination. However, in this example, we will not implement the classic pagination, but instead we will use scrolling; whenever a user scrolls down and reaches the bottom of the page, a new search will be performed with an offset of pege_number*result_per_page and, consequently, a new set of results will be rendered.

The function show_more will be called whenever the user scrolls down and hits the end of the page. Every time the function show_more is called, it increases the page number, and then calls another search function, searchMore(). The only difference between searchMore() and search() is that the latter defines an offset, .from(offset).

Highlighting

Elasticsearch implements search result highlighting on one or more fields. In the following we define highlighting on the text field, and specify that for each hit the highlighted fragments should be in bold.

The code above defines a Highlight object, then passes it to the search request. No changes are required in the search function.

Viewing, again

Now that we have defined all the functions needed to send queries to elasticsearch, we can extend our view to display search results.

The directive infinite-scroll specifies that the function show_more() should be called on scroll-down. The directive ng-repeat is like an ‘iterative loop’ that repeats the HTML template for each element in the collection specified, e.g. the directive ng-repeat="doc in results.hits.hits" iterates over all the hits returned by elastic.js from elasticsearch. Note that the array resultArr is accessible from the Twitter view because it was defined on the $scope object in TwitterController.js. renderResult and renderResultMetadata are JavaScript functions defined in TwitterController.js, both function return HTML, hence we need to sanitize their returned values using the directive ng-bind-html. We didn’t explain the functions renderResult and renderResultMetadata as they are straightforward.

In future posts, we will go through faceting with elasticsearch and Angular. If you are curious, though, the code on github already contains faceting.

Further readings

elastic.js user guide
Getting Started with elasticsearch and AngularJS: Part1 – Searching
Angular tutorial — official website
Foodme: Angular example application

Join our breakfast seminar in Oslo November 28th

Henceforth, said search server shall be known as Forage. Check it out here and update all favourites and bookmarks accordingly.

In related news, Nor.. sorry Forage is now 10% as popular as Solr on Github. Thanks to all users and contributors who are putting Forage through its paces. We love you.

This year’s summer internship program has been centered around the concept “Search driven websites”. Over the course of the summer we’ve gone from a concept and a vision to a prototype website, and we’d like to share the potential we’ve uncovered along the way with you.

What is a search driven website?

On a search driven website, everything is driven by search. More useful and less painfully obvious, a search driven website will get all its content, all its navigation and menus as a result of a search query. This allows a different and more dynamic approach to the structuring and placement of content, in contrast to a typical folder based, editor dependent website where someone has to decide exactly which articles and images will reside on the front page, or any page, at any one time.

A traditional site requires the user to spend time learning and navigating a cramped and complex menu structure to find the content desired. We believe that navigation can and should be done in a better way, through the use of an ever-ready search bar, dynamic navigators and relevant links. Even when displaying a single page, a search is performed to retrieve that article along with related results.

There are immediate advantages to using search as the main logic module of a website. Take the front page: Set the site’s opening search to “*” – search everything – and make the search engine order the results by relevance, popularity and date. Display the results as proper, readable articles, and you get a site containing the content you want to display without having to manually specify what goes where. A clear advantage here is that popular results will be displayed automatically on the front page, letting many of your readers find what they want immediately. This will also let your site adapt to your readers’ interests automatically, and not just when you perform a user test or go through a slew of analytics.

A unified experience

Within this vision of search driven websites is the idea that just as the front page content is retrieved with a search, the reader’s personal searches should also be displayed like the front page. Let’s use a specific example. Below is a screen shot from the front page of aftenposten.no, a large Norwegian news site.

This looks pretty good. Now let’s search for China and see how it looks.

Not looking so good anymore. When we perform a search, it no longer looks like a newspaper. It no longer IS a newspaper, it is a mini Google. What if, instead, the results looked just like the front page?

Now the search becomes an integrated part of the website. Readers no longer have to enter “search mode” to find what they are looking for, they simply type something into the search bar and get a site specifically tailored to their interests. It’s clean, fresh and feels natural. This part of the search driven vision could easily be implemented in most websites out there, and we believe that unifying the search and the rest of the website would make the Internet a more comfortable place to be.

Reduce maintenance and integration costs

The big advantage of search is that you can simply tell the search engine what you want and it will find the best content for you. We know this might sound obvious, but think about the consequences. Life becomes easier when all you need to do is ask the search to fill your site with content. Remember, both the site administrator and the readers can have a say in what the search engine will look for. To base your website on search can be a good alternative when it is important that relevant and updated content is displayed at all times. If your content is located in many different places and you have different readers with different information needs, a search based solution could be your best bet.

With a search based website, the decisions become all about what to search for in which situations. By gathering statistics and usage patterns, the search can be tweaked to give each reader a personalized experience. Whenever you want to add a new page to your site about a particular topic, you’d simply get the content of your new page through a customized search. The rest is just creating relevant content.

When businesses get  new software systems, merge with or acquire other businesses, you often get several sources for your content. This is an organizational pain and comes with high software integration costs. In the purest version of the search driven vision, all you would need to do is make sure your search engine can index the new source. There is no need to move your content or cross/double post articles, as long as the content is in the search engine it’ll be available through your search driven site.

Readers often want an overview of what exists on your site as they look for content. Search based navigation could be the answer. Up until recent years, people have been used to navigating through mutually exclusive folders, and this is the way many websites are built today. An alternative approach is to let the search engine figure out which categories exist, for example by observing where pages have been posted or what they are tagged with. Let clicks in the menu navigation be searches for the menu keywords, or a search for elements with a specific category tag. The real locations and structure of your content will no longer be important. This reduces the integration work on site navigation by not having to consider the actual locations of the content, just make sure the new content is categorized or tagged properly.

 Our prototype

This summer we have created a prototype for this concept, giving the rest of Comperio an example implementation and a platform for further discussions. You can  see our prototype at the address below.  To check out the category navigation click the “show categories” in the bottom right corner.

beta.comperio.no

The site is composed of four parts. A search engine, an admin interface, a search API and a client. The client is created using Javascript and handlebars.js, the search engine is an Apache Solr instance and the admin interface and search API are both contained in a wordpress plugin. The client requests content from the search API, which uses the search engine as a back end to find relevant content before it is returned to the client. Everything found in the prototype is completely search driven, including the front page, single page view, related items and category navigation. The admin interface lets you control indexing as well as different search parameters.

A last note

Over the summer we have built a website completely based on search. We believe that using search as the main content delivery method has great potential. Businesses today have vast amounts of information, and there is an ever increasing need for sharing, accessibility and flexibility. These are challenges that can’t be ignored and we have enjoyed uncovering, discussing, and hopefully getting closer to a solution to these problems in light of the concept of search driven web sites.

Norch is a search engine written for Node.js. Norch uses the Node search-index module which is in turn written using the super fast levelDB library that Google open-sourced in 2011.

The aim of Norch is to make a simple, fast search server, that requires minimal configuration to set up. Norch sacrifices complex functionality for a limited robust feature set, that can be used to set up a freetext search engine for most enterprise scenarios.

Currently Norch features

• Full text search
• Stopword removal
• Faceting
• Filtering
• Relevance weighting (tf-idf)
• Field weighting
• Paging (offset and resultset length)

Solrstrap is a Query-Result interface for Solr- it is intended to be a starting point for those building web interfaces that talk to Solr, or a very lightweight admin tool for querying Solr in a Googleish fashion.

Cool things about Solrstrap:

• Requires only local installation- easy to set up
• Access to all Bootstrap functionality. Can be easily extended in a Bootstrappy way.
• Blazing fast
• Uses less bandwidth

Use it as you see fit. Merciless criticism and fawning praise equally welcome.