Last week, I saw the future. Wohaa, that’s always a great feeling.  I’ve seen it in earlier weeks also, but now it was even brighter than before. For me, it’s still called the Lebron Stack as Max Ogden explains it and consists of LevelDB, Browserify and npm. All this is mostly happening in JavaScript. Before I’m knocked to the ground: I wasn’t the first to either make the prediction or say it out loud. I’m way behind, and it’s not a very novel or extreme idea, just a really good one. But when something is predicted, it may take a long time before it happens, if it happens at all. I think it’s happening now-ish.

So this blog post is about why I think that time is now. Disclaimer for the .Net and Java heads And all you .Net- and Java-heads will surely find some stuff that will be done better within your part of the world, but hear me out! I know the list of “This already exists in OS [W] or [X]” or “You can do that with software [X], [Y] or [Z]“. I have these thoughts my self, and I’ve been wondering why I still think that Lebron and JavaScript still will be so much more important. I’m not saying that .Net and Java stuff will go away, it will just be less important (it already is) and most of the cool and stuff closer to the user will happen in the JavaScript world.

The future is bright at the Webrebels conference in Oslo, May – 2014.

Here are the reasons I found so far

• Most stuff happens in the browser Selling anything, you want to be where the people are. For regular people that’s on their smartphone using a web app or just a native app, which in most cases is a web app wrapped as a native app. Emerging markets makes this shift towards the browser happen even faster. The Firefox OS may fail as an OS, but still succeed creating a standard smartphone API for web applications, the WebAPI. This will make it even easier to create web apps for all of the world’s smartphones, which leads me on to my next point.
• Easier for startups and developers Competing with the big ones is never easy. Amazon Web Services, AWS,  and similar services made it a little easier to scale hardware use dynamically, and from that, the cost of hardware. With the browser as a VM and single page applications a lot of the web application rendering and logic is moved from the servers to the clients. So for a small company the choice is easy. Why do all the heavy lifting on your own servers when the users can do most of the application rendering and logic on their smartphones? The irony in the old “thin vs. thick client” debate is that the clients actually got a lot thinner, and in the same go started doing more of the heavy lifting. While a Google data center is impressive, I also got a feeling it’s a sign of something gone terribly wrong.
• Collaboration, modularity and minimum effort npm is great stuff. It takes away a lot of dependency pain in the JavaScript world. Combined with people that are very good at writing small modular programs and lots of stuff under the MIT license we have a winner. We now have tools for collaboration that actually works. People build their killer apps with very little effort on top of others’ greatness. No more reinventing the text editor.
• Cheaper hardware for regular users Okay, most people access the Internet through their phone, but the Chromebook explains this point very well. Why have a full OS, with all the hardware costs to run it fairly fast, when all you do is fire up a browser? The browser is the OS more and more each day. Last time my desktop at home broke down, I bought a new one. The new one was state of the art and it was a miscalculation buying it. Almost every time I boot it (running Ubuntu), I’m asked to upgrade to the newest version. That means every half year or so. The laptop I have, I actually use a little, but much less than my pad/tablet and phone.
• Everything fun is online Not a real argument, but hey… Isn’t it true?

But what about the Forage search engine you say?

So, what does these reasons for Lebron/JavaScript’s future success have to do with the Forage search engine? First of all, it’s written in JavaScript and needs very little hardware to run properly. You install it with npm, and that takes care of all the dependencies, like LevelDB, where the data is actually stored. Hopefully it will run in the browser in near future using Browserify and make testing, installing and maintaining search software so much easier and more accessible. It also opens up a lot of new interesting use cases for search. My guess is that it won’t compete with the bigger search engines, but that it will open up the possibility for better and cheaper search functionality for small scale solutions.

Anything you want to add to or subtract from the list?

Crawl interfaces, suggested

Initial Crawl-window

To crawl most pages elegantly and easily, you need five information elements:

1. Somewhere to start. Which place do you want your crawler to start. You don’t have to specify the domain, we pick the domain name  from the page you’re visiting.
2. Which links to follow. This is not necessarily the pages you want to crawl. Typically these pages have lists of pages you want to crawl.
3. Which links not to follow. To not make the crawler go wild, you set some boundaries. Often a page has several URLs.
4. Which links to crawl. These are the actual pages you’re looking for.
5. Which links not to crawl.

A simple illustration on the above rules. Forage Fetch doesn’t have all these features yet, but they’re suggested as enhancements.

Selecting which rule type to add

To ensure you’re adding valid rules, it’s a good ting to test first.

Start URL added

The minimum amount of rules needed to start the crawler

Next tasks will be to make a clickable prototype in HTML/CSS and read up on HTML5 local storage/web storage.

All comments on the idea are welcome! Here’s what we’ve blogged about Forage so far.

Forage could then be added with a bookmarklet to any page (A bookmark adding a javascript to the page you’re on). With some simple UI-stuff you could define the Forage Document Processor Adapter, set up rules for Forage Crawler, crawl, process, index and then search within your indexed documents. All without using any servers, on premises or in the cloud. When the user is satisfied she or he could download the JSON-file with processed documents + scripts for adding a search box,  search result and navigators to a page.

Possible use cases for search server running inside browser:

• Easy site search setup
  One real benefit, and the initial idea, would be that the user would not need any server to test Forage and actually crawl a site. When page crawled the user can download JSON ready to be indexed + setup-files for a search box, navigators and search result. Or add it to a cloud service and there continue the work you started in your browser.
• A easy and modern search solution behind the firewall
  Behind the firewall, almost all software looks a bit duller, more beige and basically not modern. But through the browser you could easily combine the strength of  Forage and all the hidden gems behind a firewall. There would be some big issues with security, but for intranet and people search it could be a great solution.
• Ad hoc search on a site that is not yours
  Say you’re looking for something on a site. How about ad hock index it and then search it. Yes, it’s a bandwidth abuse waiting to happen, but could make a good tool for a lot of situations.
• Your life, searchable
  This may need a browser add on, but then again, maybe not. Anyway: How about your whole online life, searchable. Today you have your browser history. It shows you page title and page link. What if all the text and images was searchable?

Some UX sketches of the idea:

The user finds a page to crawl …

… clicks the bookmarklet …

... that adds Forage JavaScript-stuff to the page …

… much like a browser plugin or add on …

… tests a jQuery selector statement …

… and adds the field to the item when satisfied. Repeated until a full item is defined.

Here’s the feature suggestion at the Forage GitHub page. Ideas or comments are more than welcome! Want to know more about Forage? Check out the Forage GitHub-pages or stuff we’ve written about Forage.

EDIT: Drawn some new mock ups on the crawler part: Forage Fetch and written about the killer combo Lebron and what it will mean for search.

1. Learn more about search, both tech and UX.
2. Show that a nice search user experience doesn’t need a search input box.
3. Show a search that is light weight.
4. Generally, build something nice, quickly, to show off.

“But what is it?” you ask again. The second answer is that it’s a search application for recipes containing the most in-season vegetables, fruits, mushrooms, fish, shellfish, poultry and meat. So, a search without a search box giving you the recipes that will make you pick the freshest, most tasty food available at any given time of the year. In Norway, you can get strawberries the whole year around, but it’s only a month or two during the summer that they taste really good. and are cheap. This goes for a lot of different foods.

We’ll blog every step of the journey. First small hackathon is this Friday, January 7th. To test our ideas we’ve already crawled food recipes from Klikk.no, and transformed them into JSON-format by using Forage Document Processor from Fergus McDowell’s excellent Forage Search Engine.

Next step is to get an Amazon EC2-server up and running, install Elasticsearch, and feed it recipes in JSON-format. We’ll also make some rough mockups of how the user interface is going to be, and maybe, hopefully get a simple version of the Seasonal Food Recipe Web Application up and running. If we after a while get enough traffic, we’ll try to do some machine learning. And we’ll definitely test out different ways of doing relevancy tuning.

Sounds nice? This is work in progress, so check back every now and then for new blog posts.

0.3.0 fixes lots niggles with indexing, and gives a pretty hefty improvement to memory usage. There is also a built in matcher for creating cool auto-suggest and auto-complete controls based on the content of the corpus.

In related news there is now a family of crawling tools which allow you to scrape, process and index web content into your Forage server. Check out forage-fetch, forage-document-processor and forage-indexer on the forage.js GitHub group.

As always, feedback, pull requests, comments, praise, criticism and beer are most welcome.

Henceforth, said search server shall be known as Forage. Check it out here and update all favourites and bookmarks accordingly.

In related news, Nor.. sorry Forage is now 10% as popular as Solr on Github. Thanks to all users and contributors who are putting Forage through its paces. We love you.

Norch is a search engine written for Node.js. Norch uses the Node search-index module which is in turn written using the super fast levelDB library that Google open-sourced in 2011.

The aim of Norch is to make a simple, fast search server, that requires minimal configuration to set up. Norch sacrifices complex functionality for a limited robust feature set, that can be used to set up a freetext search engine for most enterprise scenarios.

Currently Norch features

• Full text search
• Stopword removal
• Faceting
• Filtering
• Relevance weighting (tf-idf)
• Field weighting
• Paging (offset and resultset length)