Crawl interfaces for Forage running inside your browser

Got an idea a while back on how we could use the JavaScript/Nodejs Search Engine Forage so that the users would have their own search server inside the browser. The main takeaway from this would be that you don’t need to install anything to test the search engine. Since last time, I’ve made a quick logo for Forage, and drawn some more user interfaces. The mockups are mainly about crawl interfaces setting up the crawler, which in Forage terms is called Forage Fetch.

Crawl interfaces, suggested

Initial Crawl-window
javascript crawl interfaces

To crawl most pages elegantly and easily, you need five information elements:

  1. Somewhere to start. Which place do you want your crawler to start. You don’t have to specify the domain, we pick the domain name  from the page you’re visiting.
  2. Which links to follow. This is not necessarily the pages you want to crawl. Typically these pages have lists of pages you want to crawl.
  3. Which links not to follow. To not make the crawler go wild, you set some boundaries. Often a page has several URLs.
  4. Which links to crawl. These are the actual pages you’re looking for.
  5. Which links not to crawl.

A simple illustration on the above rules. Forage Fetch doesn’t have all these features yet, but they’re suggested as enhancements.

Selecting which rule type to add
javascript crawl interfaces

To ensure you’re adding valid rules, it’s a good ting to test first.
javascript crawl interfaces

Start URL added
javascript crawl interfaces

The minimum amount of rules needed to start the crawler
javascript crawl interfaces

Next tasks will be to make a clickable prototype in HTML/CSS and read up on HTML5 local storage/web storage.

All comments on the idea are welcome! Here’s what we’ve blogged about Forage so far.

Article written by

Espen Klem
Interaction designer with a love for log reading, statistics and mind-bending, user friendly concepts.

1 response to: «Crawl interfaces for Forage running inside your browser»

  1. [...] Posted earlier on Search Nuggets. [...]

Leave a response

XHTML: These tags are allowed: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code class="" title="" data-url=""> <del datetime=""> <em> <i> <q cite=""> <strike> <strong> <pre class="" title="" data-url=""> <span class="" title="" data-url="">

301 Moved Permanently

Moved Permanently

The document has moved here.


Comperio AS
Øvre Slottsgate 27
NO-0157 Oslo,
+47 22 33 71 00
View map


Search Provider Sverige AB
Gamla Brogatan 34
SE-11 120 Stockholm
+46 8-21 49 00
View map