Men medans vi hyllar det enkla användargränssnittet så önskar vi att det gick att bygga bättre sökfunktionalitet. Visst kan du söka i en webbsajt, det går även att söka på andra webbsajter, om du har satt upp flera webbsajter inom samma system. Men om du vill söka i en webbsajt eller databas som finns någon annanstans då går det inte. Men detta är på väg att ändras. Sitevision introducerar snart webbkravlaren Nutch, en mycket avancerad webcrawler som bygger på Hadoop, som i sin tur är del av ett ramverk för att hantera mycket stora mängder data. Nutch tillsammans med Solr kommer att lyfta Sitevisions sök till nya höjder.

Nedan är ett schema för hur en sajt-indexering skulle kunna se ut:

1. ”Injector” tar alla webbadresser i nutch.txt filen och lägger till dem i ”CrawlDB”. Som är en central del av Nutch. CrawlDB innehåller information om alla kända webbadresser (hämta schema, hämta status, metadata, …).

2. Baserat på data från CrawlDB skapar ”Generator” en lista på vad som ska hämtas och placerar det i en nyskapad segment katalog.

3. Nästa steg, ”Fetcher” får de adresser som ska hämtas från listan och skriver det tillbaka till segment katalogen. Detta steg är vanligtvis den mest tidskrävande delen.

4. Nu kan ”Parser” behandla innehållet i varje webbsida och exempelvis utelämnar alla html-taggar. Om denna hämtning (crawl) är en uppdatering eller en utökning av en redan tidigare hämtning (t.ex. djup 3), skulle ”Updater” lägga till nya data till CrawlDB som ett nästa steg.

5. Före indexering måste alla länkar inverteras av ”Link Inverter”, som tar hänsyn till att inte antalet utgående länkar på en webbsida är av intresse, utan snarare antalet inkommande länkar. Detta är ganska likt hur Google Pagerank fungerar och är viktig för scoring funktion. De inverterade länkarna sparas i “Linkdb”.

6-7. Med hjälp av data från alla möjliga källor (CrawlDB, LinkDB och segment), skapar Indexer ett index och sparar det i Solr katalogen. För indexering, används det populära Lucene biblioteket. Nu kan användaren söka efter information om genomsökta webbsidor via Solr.

Funktionalitet som följer med Nutch:

• Indexering av externa källor
• Automatisk kategorisering
• Metadata
• Textanalys
• Utökad funktionalitet enkelt med plugins

Nyttiga länkar:

• https://en.wikipedia.org/wiki/Apache_Nutch
• http://wiki.apache.org/nutch/
• http://nutch.apache.org/
• http://www.sitevision.se/vara-produkter/sitevision.html

Although I usually write about and work with Apache Solr, I also use the ELK stack on a daily basis on a number of projects. If you’re not familiar with Solr, take a look at some of my previous posts. If you need some more background info on the ELK stack, both Christoffer and André have written many great posts on various ELK subjects. The most common use for the stack is data analysis. In our case, Solr search log analysis.

As a little side note for the truly devoted Solr users, an ELK stack alternative exists with SiLK. I highly recommend checking out Lucidworks’ various blog posts on Solr and search in general.

Some background

On an existing search project I use the ELK stack to ingest, analysis and visualise logs from Comperio’s search middleware application.

Although this gave us a great view of user query behaviour, Solr logs a great more detailed information. I wanted to log indexing events, errors and the searches with all other parameters in addition to just the query string.

Lets get started

I’m going to assume you already have a running Solr installation. You will, however, need to download Elasticsearch and Logstash and unpack them. Before we start Elasticsearch, I recommend installing these plugins:

• Head
• Marvel

Head is a cluster health monitoring tool. Marvel we’ll only need for the bundled developer console, Sense. To disable Marvel’s other capabilities, add this line to ~/elasticsearch/config/elasticsearch.yml

marvel.agent.enabled: false

Start elasticsearch with this command:

~/elasticsearch-[version]/bin/elasticsearch

Navigate to http://localhost:9200/ to confirm that Elasticsearch is running. Check http://localhost:9200/_plugin/head and http://localhost:9200/_plugin/marvel/sense/index.html to verify the plugins installed correctly.

The anatomy of a Logstash config

Update 21/09/15

I have since greatly simplified the multiline portions of the Logstash configs. Use instead this filter section: The rest of the original article contents are unchanged for comparison’s sake.

All Logstash configs share three main building blocks. It starts with the Input stage, which defines what the data source is and how to access it. Next is the Filter stage, which carries out data processing and extraction. Finally, the Output stage tells Logstash where to send the processed data. Lets start with the basics, the input and output stages:

input {
  file {
    path => "~/solr.log"
  }
}

filter {}

output {
  # Send directly to local Elasticsearch
  elasticsearch_http {
    host => "localhost"
    template => "~/logstash/bin/logstash_solr_template.json"
    index => "solr-%{+YYYY.MM.dd}"
    template_overwrite => true
  }

This is one of the simpler input/output configs. We read a file at a given location and stream its raw contents to an Elasticsearch instance. Take a look at the input and output plugins’ documentation for more details and default values. The index setting causes Logstash to create a new index every day with a name generated from the provided pattern. The template option tells Logstash what kind of field mapping and settings to use when creating the Elasticsearch indices. You can find the template file I used here.

To process the Solr logs, we’ll use the grok, mutate, multiline, drop and kv filter plugins.

• Grok is a regexp based parsing stage primarily used to match strings and extract parts. There are a number of default patterns described on the grok documentation page. While building your grok expressions, the grok debugger app is particularly helpful. Be mindful though that some of the escaping syntax isn’t always the same in the app as what the Logstash config expects.
• We need the multiline plugin to link stacktraces to their initial error message.
• The kv, aka key value, plugin will help us extract the parameters from Solr indexing and search events
• We use mutate to add and remove tags along the way.
• And finally, drop to drop any events we don’t want to keep.

The hard fun part

Lets dive into the filter stage now. Take a look at the config file I’m using. The Grok patterns may appear a bit daunting, especially if you’re not very familiar with regexp and the default Grok patterns, but don’t worry! Lets break it down.

The first section extracts the log event’s severity and timestamp into their own fields, ‘level’ and ‘LogTime’:

grok {
    match => { "message" => "%{WORD:level}.+?- %{DATA:LogTime};" }
      tag_on_failure => []
  }

So, given this line from my example log file

INFO  - 2015-09-07 15:40:34.535; org.apache.solr.update.processor.LogUpdateProcessor; [sintef_main] webapp=/ path=/update/extract params={literal.source=epifile&literal.epi_file_title=GOFER+L4.0+Demonstratorer+V1.0.pdf&literal.title=GOFER+L4.0+Demonstratorer+V1.0.pdf&literal.id=epifile_211278&literal.epifileid_s=211278&literal.url=http://www.sintef.no/globalassets/upload/teknologi_samfunn/6060/prosjektfiler/gofer/gofer-l4.0-demonstratorer-v1.0.pdf&stream.url=http://www.sintef.no/globalassets/upload/teknologi_samfunn/6060/prosjektfiler/gofer/gofer-l4.0-demonstratorer-v1.0.pdf&literal.filesource_s=SiteFile} {} 0 65

We’d extract

{ "level": "INFO", "LogTime":"2015-09-07 15:40:34.535"}

In the template file I linked earlier, you’ll notice configuration for the LogTime field. Here we define for Elasticsearch a valid DateTime format. We need to do this so that Kibana recognises the field as one we can use for temporal analyses. Otherwise the only timestamp field we’d have would contain the time at which the logs were processed and stored in Elasticsearch. Although not a problem in a realtime log analyses system, if you have old logs you want to parse you will need to define this separate timestamp field. As an additional sidenote, you’ll notice I use

tag_on_failure => []

in most of my Grok stages. The default value is “_grokparsefailure”, which I don’t need in a production system. Custom failure and success tags are very helpful to debug your Logstash configs.

The next little section combines commit messages into a single line. The first event in the example log file is an example of such commit messages split over three lines.

# Combine commit events into single message
  multiline {
      pattern => "^\t(commit\{)"
      what => "previous"
    }

Now we come to a major section for handling general INFO level messages.

# INFO level events treated differently than ERROR
  if "INFO" in [level] {
    grok {
      match => {
          "message" => ".+?; ((([a-zA-Z]+(\.|;|:))+) )+?\[%{WORD:collection}\].+?path=%{DATA:endpoint} params=\{%{DATA:params}\}.+?\{%{WORD:action}=\[%{DATA:docId}"
        }
        tag_on_failure => []  
    }
    if [params] {
      kv {
        field_split => "&"
        source => "params"
      }
    } else {
      grok {
        match => {
          "message" => ".+?; ((([a-zA-Z]+(\.|;|:))+) )+?commits"  
        }
        tag_on_failure => [ "drop" ]
        add_field => {
          "action" => "commit"
        }
      }
      if "drop" in [tags] {
        drop {}
      }
    }
  }

This filter will only run on INFO level messages, due to the conditional at its beginning. The first Grok stage matches log events similar to the one above. The key fields we extract are the Solr collection/core, the endpoint we hit e.g. update/extract, the parameters supplied by the HTTP request, what action e.g. add or delete and finally the document ID. If the Grok succeeded to extract a params field, we run the key value stage, splitting no ampersand to extract each HTTP parameter. This is how a resulting document’s extracted contents look like when stored in Elasticsearch:

{
  "level": "INFO",
  "LogTime": "2015-09-07 15:40:18.938",
  "collection": "sintef_main",
  "endpoint": "/update/extract",
  "params":     "literal.source=epifile&literal.epi_file_title=A05100_Tass5+Trondheim.pdf&literal.title=A05100_Tass5+Trondheim.pdf&literal.id=epifile_211027&literal.epifileid_s=211027&literal.url=http://www.sintef.no/globalassets/upload/teknologi_samfunn/5036/a05100_tass5-trondheim.pdf&stream.url=http://www.sintef.no/globalassets/upload/teknologi_samfunn/5036/a05100_tass5-trondheim.pdf&literal.filesource_s=SiteFile",
  "action": "add",
  "docId": "epifile_211027",
  "version": "1511661994131849216",
  "literal.source": "epifile",
  "literal.epi_file_title": "A05100_Tass5+Trondheim.pdf",
  "literal.title": "A05100_Tass5+Trondheim.pdf",
  "literal.id": "epifile_211027",
  "literal.epifileid_s": "211027",
  "literal.url": "http://www.sintef.no/globalassets/upload/teknologi_samfunn/5036/a05100_tass5-trondheim.pdf",
  "stream.url": "http://www.sintef.no/globalassets/upload/teknologi_samfunn/5036/a05100_tass5-trondheim.pdf",
  "literal.filesource_s": "SiteFile"
}

If the Grok did not extract a params field, I want to identify possible commit messages with the following Grok. If this one fails we tag messages with “drop”. Finally, any messages tagged with “drop” are dropped for the pipeline. I specifically created these Grok patterns to match indexing and commit messages as I already track queries at the middleware layer in our stack. If you want to track queries at the Solr level, simply use this pattern:

.+?; ((([a-zA-Z]+(\.|;|:))+) )+?\[%{WORD:collection}\].+?path=%{DATA:endpoint} params=\{%{DATA:params}\} hits=%{INT:hits} status=%{INT:status} QTime=%{INT:queryTime}

The next section handles ERROR level messages:

# Error event implies stack track, which requires multiline parsing
  if "ERROR" in [level] {
    multiline {
      pattern => "^\s"
      what => "previous"
      add_tag => [ "multiline_pre" ]
    }
    multiline {
        pattern => "^Caused by"
        what => "previous"
        add_tag => [ "multiline_post" ]
    }
    if "multiline_post" in [tags] {
      grok {
        match => {
          "message" => ".+?; ((([a-zA-Z]+(\.|;|:))+) )+%{DATA:reason}(\n\t)((.+?Caused by: ((([a-zA-Z]+(\.|;|:))+) )+)%{DATA:reason}(\n\t))+"
        }
        tag_on_failure => []
      }
    }
  }

Given a stack trace, there’s a few in the example log file, this stage first combines all the lines of the stack trace into a single message. It then extracts the first and the last causes. The assumption being the first message is the high level failure message and the last one the actual underlying cause.

Finally, I drop any empty lines and clean up temporary tags:

# Remove intermediate tags, and multiline added randomly by multiline stage
  mutate {
      remove_tag => [ "multiline_pre", "multiline_post", "multiline" ]
  }
  # Drop empty lines
  if [message] =~ /^\s*$/ {
    drop {}
  }

To check you have succesfully processed your Solr logs, open up the Sense plugin and run this query:

# aggregate on level
GET solr-*/_search
{
  "query": {
    "match_all": {}
  },
  "size": 10,
  "aggs": {
    "action": {
      "terms": {
        "field": "level",
        "size": 10
      }
    }
  }
}

You should get back all your processed log events along with an aggregation on event severity.

Conclusion

Solr logs contain a great deal of useful information. With the ELK stack you can extract, store, analyse and visualise this data. I hope I’ve given you some helpful tips on how to start doing so! If you run into any problems, please get in touch in the comments below.

Stream processing, Internet of things, Real time analytics, Big data, Recommendations, Machine learning. Berlin Buzzwords undoubtedly lives up to its name by presenting the frontlines of data technology trends.

The conference is focused on three core concepts – search, data and scale, bringing together a diverse range of people and with presentations touching the perimeter of the buzzword range.
Berlin Buzzwords kicked off on Sunday evening with a Barcamp, Monday and Tuesday contained full day conferences, while Wednesday was filled with hackathons and workshops.

Comperio

Comperio was one of the many companies sponsoring the conference, and came to Berlin bringing two speakers. André Lynum talked about “Beyond Significant terms” – a deep dive into how to utilize Elasticsearch built in indexes and APIs for improved lexical analysis, topic management and trend information. André’s talk went far beyond what the well known Elasticsearch significant terms aggregation provides. Christoffer Vig captured a spot on the informal Open Stage, giving a funny and off-kilter presentation and demo of the analytics and visualization capabilities of Kibana 4 based on a beer product catalogue.

The talks

Many people attended the comparison of Solr and Elasticsearch Performance & Scalability with Radu Gheorghe & Rafał Kuć from Sematext. This was a fast paced run through of how they were able to create tests reproducing the same conditions on both search engines. Elasticsearch outperformed Solr on text search using wikipedia data, while, surprisingly Solr outperformed Elasticsearch on aggregations. Solr has recently started catching up with Elasticsearch on providing nested aggregations and perhaps the improved performance comes as a result of a slimmed down implementation? It will be very interesting to follow the developments of both platforms into the future, and as consumers of the products we see competition is a good thing driving innovation and performance.

Two other interesting technical talks was Adrian Grands explaining some of the algorithms behind Elasticsearchs aggregations and Ted Dunnings presentation of the t-digest algorithm. Both were a window into how approximations can yield fast algorithms for complex statistics with provable bounds which they managed to keep approachable to the casual listener.

SQL?

Another theme threatening to return from the basement was how to properly support SQL style joins into search engines. Real life use cases sometimes demand objects with relations. The stock answer from the NoSQL world is to denormalize your data before inserting it, but Lucene/Elasticsearch/Solr did get limited Join support a while ago. Taking this further Mikhail Khludnev showed how the new Global Ordinal Join aims to provide a Join with improved performance.

Talking the talk

As search consultants one of our main challenges at Comperio is communicating about technical topics with customers who need to connect technical topics to their own competence and background. Ellen Friedman from MapR explained how such communication can be beneficial to almost any team or team member and shared some experiences and ideas regarding how you can try this at home. At its core it boils down to understanding and describing your technical work across several layers and showing respect for the perspective and background your conversation partner.
She also shared a very funny parrot joke. Not going to reveal that one here, watch the video if you’ld like a good laugh.

Hackathon

Comperio also attended the Apache Flink workshop hosted at Google’s offices in Berlin by the talented developers at data Artisans. Apache Flink is in some ways similar to Apache Spark and other recent distributed computing frameworks, and is an alternative to Hadoop’s MapReduce component. It represents a novel approach to data processing, modelling all data as streams, exposing both a batch- and stream APIs. Apache Flink has a built in optimizer that optimizes memory, network traffic and processing power. This leaves the developer to implement core functionality in Java, Scala or Python.

The buzz

Berlin Buzzwords is a great opportunity to surf the crest of the big data wave with the most interesting people in the field. The city of Berlin with it’s sense of being on the edge of new developments provides the perfect backdrop for a conference on the latest “Buzzwords”. Comperio will certainly be back next year.

Videos from most talks are available at youtube.com

Beyond significant terms

Algorithms and data-structures that power Lucene and Elasticsearch

Practical t-digest Applications

Talk the Talk: How to Communicate with the Non-Coder

Side by Side with Elasticsearch & Solr part 2

• Facets
• Spellcheck
• Phonetic search
• Query Completion

Setting up our environment

The requirements remain the same as with the original blogpost:

1. Java 1.7 or greater
2. A MySQL database
3. A copy of the sample employees database
4. The MySQL jdbc driver

We’ll now be using Solr 5, which runs a little differently from previous incarnations of Solr. Download Solr and extract it to a directory of your choice.Open a terminal and navigate to your Solr directory.
Start Solr with the command

bin/solr start

To confirm Solr successfully started up, run 

bin/solr status

Unlike previously, we now need to create a Solr core for our employee data. To do so run this command

bin/solr create_core -c employees -d basic_configs

bin/solr create_core -help

1. Open server/solr/employees/conf/solrconfig.xml in a text editor and add the following within the config tags:

   <lib dir="../../../dist/" regex="solr-dataimporthandler-\d.*\.jar" />
    
   <requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
   <lst name="defaults">
   <str name="config">db-data-config.xml</str>
   </lst>
   </requestHandler>

2. In the same directory, open schema.xml and add this this line:
   

   <dynamicField name="*_name" type="text_general" multiValued="false" indexed="true" stored="true" />

3. Create a lib subdir in server/solr/employees and extract the MySQL jdbc driver jar into it.
4. Finally, restart the Solr server with the command 

   bin/solr restart

When started this way, Solr runs by default on port 8983. Use

bin/solr start -p portnumber

Navigate to http://localhost:8983/solr and you should see the Solr admin GUI splash page. From here, use the Core Selector dropdown button to select our employee core and then click on the Dataimport option. Expanding the Configuration section should show an XML response with a stacktrace with a message along the lines of

Can't find resource 'db-data-config.xml' in classpath

We’ll come back to that file later but let’s make our demo database now. If you haven’t already downloaded the sample employees database and installed MySQL, now would be a good time!

Setting up our database

Please refer to the instructions in the same section in the original blog post. The steps are still the same.

Indexing our database

Again, please refer to the instructions in the same section in the original blog post. The only difference is the Postman collection should be imported from this url instead. The commands you can use alternatively have also changed and are now

Clear index: http://localhost:8983/solr/employees/update?stream.body=<delete><query>*:*</query></delete>&commit=true
Retrieve all: http://localhost:8983/solr/employees/select?q=*:*&omitHeader=true
Index db: http://localhost:8983/solr/employees/collection1/dataimport?command=full-import
Reload core: http://localhost:8983/solr/employees/admin/cores?action=RELOAD&core=collection1
Georgi query: http://localhost:8983/solr/employees/select?q=georgi&wt=json&qf=first_name%20last_name&defType=edismax
Facet query: http://localhost:8983/solr/employees/select?q=*:*&wt=json&facet=true&facet.field=dept_s&facet.field=title_s&facet.mincount=1&rows=0
Gorgi spellcheck: http://localhost:8983/solr/employees/select?q=gorgi&wt=json&qf=first_name&defType=edismax
Georgi Phonetic: http://localhost:8983/solr/employees/select?q=georgi&wt=json&qf=first_name%20last_name%20phonetic&defType=edismax

The next step

We should now be back where we ended with the original blog post. So far we have successfully

• Setup a database with content
• Indexed the database into our Solr index
• Setup basic scheduled delta reindexing

Let’s get started with the more interesting stuff!

Facets

Facets, also known as filters or navigators, allow a search user to refine and drill down through search results. Before we get started with them, we need to update our data import configuration. Replace the contents of our existing db-data-config.xml with:

select e.emp_no as 'id', e.birth_date,
(
select t.title
order by t.`from_date` desc
limit 1
) as 'title_s', e.first_name, e.last_name, e.gender as 'gender_s', d.`dept_name` as 'dept_s'
from employees e
join dept_emp de on de.`emp_no` = e.`emp_no`
join departments d on d.`dept_no` = de.`dept_no`
join titles t on t.`emp_no` = e.`emp_no`
group by e.`emp_no`
limit 1000;

You’ll notice we map title, gender and dept_name to title_s, gender_s and dept_s respectively. This allows us to take advantage of an existing dynamic field mapping in Solr’s default basic config, *_s. A dynamic field allows us to assign all fields with a certain pre/suffix the same field type. In this case, given the field type

<dynamicField name="*_s" type="string" indexed="true" stored="true" />

1. Clear the index and restart Solr.
2. Once Solr has restarted, reindex the database with
   our new SQL. Don’t be alarmed if this takes a bit longer
   than previously. It’s a bit more heavy weight and not
   very well optimised!
3. Once it’s done indexing, we can
   confirm it was successful by running the facet query via
   Postman or directly in our browser.
4. We should see two hits for the query “georgi” along with
   facets for their respective titles and department.

The anatomy of a facet query

Let’s take a closer look at the relevant request parameters of our facet query: 

http://localhost:8983/solr/employees/select?q=georgi&wt=json&qf=first_name%20last_name&defType=edismax&omitHeader=true&facet=true&facet.field=dept_s&facet.field=title_s&facet.mincount=1

• facet – Tells Solr to enable or prevent faceting. Accepted values include yes,on and true to enable, no, off and false to disable
• facet.field – Which field we want to facet on, can be defined multiple times
• facet.mincount – The minimum number of values for a particular facet value the query results includes for it to be included in the facet result object. Can be defined per facet field with this syntax f.fieldName.facet.mincount=1

There are many other facet parameters. I recommend taking a look at the Solr wiki pages on faceting and other possible parameters.

Spellcheck

Analysing query logs and focusing on those queries that gave zero hits is a quick and easy way to see what can and should be done to improve your search solution. More often than not you will come across a great deal of spelling errors. Adding spellcheck to a search solution gives such great value for a tiny bit of effort. This fruit is so low hanging it should hit you in the face!

To enable spellcheck, we need to make some configuration changes.

1. In our schema.xml, add these two lines after the *_name dynamic field type we added earlier:

   <copyField source="*_name" dest="spellcheck" />
   <field name="spellcheck" type="text_general" indexed="true" stored="true" multiValued="true" />

   A copyField checks for fields whose names match the pattern defined in source and copies their destinations to the dest field. In our case, we will copy content from first_name and last_name to spellcheck. We then define the spellcheck field as multiValued to handle its multiple sources.

2. Add the following to our solrconfig.xml:

   <searchComponent name="spellcheck" class="solr.SpellCheckComponent">
   <str name="queryAnalyzerFieldType">text_general</str>
   <!-- a spellchecker built from a field of the main index -->
   <lst name="spellchecker">
   <str name="name">default</str>
   <str name="field">spellcheck</str>
   <str name="classname">solr.DirectSolrSpellChecker</str>
   <!-- the spellcheck distance measure used, the default is the internal levenshtein -->
   <str name="distanceMeasure">internal</str>
   <!-- minimum accuracy needed to be considered a valid spellcheck suggestion -->
   <float name="accuracy">0.5</float>
   <!-- the maximum #edits we consider when enumerating terms: can be 1 or 2 -->
   <int name="maxEdits">2</int>
   <!-- the minimum shared prefix when enumerating terms -->
   <int name="minPrefix">1</int>
   <!-- maximum number of inspections per result. -->
   <int name="maxInspections">5</int>
   <!-- minimum length of a query term to be considered for correction -->
   <int name="minQueryLength">4</int>
   <!-- maximum threshold of documents a query term can appear to be considered for correction -->
   <float name="maxQueryFrequency">0.01</float>
   <!-- uncomment this to require suggestions to occur in 1% of the documents
   <float name="thresholdTokenFrequency">.01</float>
   -->
   </lst>
   </searchComponent>

   This will create a spellchecker component that uses the spellcheck field as its dictionary source. The spellcheck field contains content copied from both first and last name fields.

3. In the same file, look for the select requestHandler and update it to include the spellcheck component:

   <requestHandler name="/select" class="solr.SearchHandler">
   <!-- default values for query parameters can be specified, these
   will be overridden by parameters in the request
   -->
   <lst name="defaults">
   <str name="echoParams">explicit</str>
   <int name="rows">10</int>
   <str name="spellcheck">on</str>
   <str name="spellcheck.dictionary">default</str>
   </lst>
   <!-- Add this to enable spellcheck -->
   <arr name="last-components">
   <str>spellcheck</str>
   </arr>
   </requestHandler>

The defaults list in a requestHandler defines which default parameters to add to each request made using the chosen request handler. You could, for example, define which fields to query. In this case we’re enabling spellcheck and using the default dictionary as defined in our solrconfig.xml. All values in the defaults list can be overwritten per request. To include request parameters that cannot be overwritten, we would need to use an invariants list instead:

<lst name="invariants">
<str name="defType">edismax</str>
</lst>

Both lists can be used simultaneously. When duplicate keys are present the values in the invariants list will take precedence.

Once we’ve made all our configuration changes, let’s restart Solr and reindex. To verify the changes worked, do a basic retrieve all query and check the resulting documents for the spellcheck field. Its contents should be the same as the document’s first_name and last_name fields.

Because we have enabled spellcheck by default, queries with possible suggestions will include contents in the spellcheck response object.

Try the Gorgi spellcheck query and experiment with different queries. To query the last_name field as well, change the qf parameter to 

qf=first_name last_name

The qf parameter defines which fields to use as the search domain.

When the spellcheck response object has content, you can easily use it to implement a basic “did you mean” feature. This will vastly improve your zero hit page.

Phonetic Search

Now that we have a basic spellcheck component in place, the next best feature that easily creates value in a people search system is phonetics. Solr ships with some basic phonetic tokenisers. The most commonly used out of the box phonetic tokeniser is the DoubleMetaphoneFilterFactory. It will suffice for most use cases. It does, however, have some weaknesses, which we will go into briefly in the next section.

We need to once again modify our schema.xml to take advantage of Solr’s phonetic capabilities. Add the following:

<fieldType name="phonetic" class="solr.TextField" >
 <analyzer>
 <tokenizer class="solr.StandardTokenizerFactory"/>
 <filter class="solr.DoubleMetaphoneFilterFactory" inject="true" maxCodeLength="4"/>
 </analyzer>
 </fieldType>

 <copyField source="*_name" dest="phonetic" />
 <field name="phonetic" type="phonetic" indexed="true" stored="false" multiValued="true" />

Similar to spellcheck, we copy contents from the name fields into a phonetic field. Here we define a phonetic field, whose values will not be stored as we don’t need to return them in search results. It is, however, indexed so we can actually include it in the search domain. Finally, like spellcheck, it is multivalued to handle multiple potential sources. The reason we create an additional search field is so we can apply different weightings to exact matches and phonetic matches.

Restart Solr, clear the index and reindex.

Running the Georgi Phonetic search request should now returns hits based on exact and phonetic matches. To ensure that exact matches are ranked higher, we can add a query time boost to our query fields: 

&qf=first_name last_name phonetic^0.5

Rather than apply boosts to fields we want to rank higher, it’s usually simpler to apply a punitive boost to fields we wish to rank lower. Replace the qf parameter in the Georgi Phonetic request and see how the first few results all have an exact match for georgi in the first_name field.

Query Analysis

As we look further down the result set, you will notice some strange matches. One employee, called Kirk Kalsbeek, is apparently a match for “georgi”. To understand why this is a match, we can use Solr’s analysis tool.

It allows use to define an indexed value, a query value and the field type to use and then demonstrate how each value is tokenised and whether or not the query would result in a match.

With the values Kirk Kalsbeek, georgi and phonetic respectively, the analysis tool shows us that Kirk gets tokenised to KRK by our phonetic field type. Georgi is also tokenised to KRK, which results in a match.

To create a better phonetic search solution, we would have to implement a custom phonetic tokeniser. I came across an example, which has helped me enormously in improving phonetic search for Norwegian names on a project.

Conclusion

We should now be able to

• Implement index field based spellcheck
• Use basic faceting
• Implement Solr’s out of the box phonetic capabilities

Query completion I will leave for the next time. I promise you won’t have to wait as long between posts as last time :)

Let me know how you get on in the comments below!

A bit of Solr background

Solr ships with all the tools and features necessary for an advanced search solution. These include the oft overlooked update request processors. They operate at the document level i.e. prior to individual field tokenisation and allow you to clean, modify and/or enrich incoming documents. Processing options include language identification, duplicate detection and HTML markup handling. Create a chain of them and you have a true document processing pipeline.

The Solr wiki includes a brief entry on the topic with an example of a custom processor that conditionally adds the field “cat” with value “popular”. The full list of UpdateRequestProcessor factories is available via the Solr Start project.

Back to the initial request

Certain incoming documents would contain a field, topicRef for example, with a reference to one or more documents already present in the index. The referenced documents could either contain a subsequent reference or content that we wanted to add to the incoming document.

I needed a mechanism to retrieve any referenced documents, traverse a tree of subsequently referenced documents if necessary, and then map the eventual leaf documents’ specified content fields to additional new fields in the incoming document.

I created a recursive document enrichment processor to do just that!

Its settings allow for multiple potential field retrievals and mappings, local and foreign key field definitions and the option to retrieve content from a remote Solr index.

A minor drawback of the current iteration of the processor is a high reliance on the existence of referenced documents i.e. if the referenced documents are not already present in the index then the processor will skip over them. To ensure documents are fully enriched, especially if the referenced documents are included in the same indexing batch, reindexes of incoming documents is necessary unless explicitly defining the document indexing order.

In addition, when a referenced document is updated, content owners expect this to have an impact on the content of the parent document and therefore a user’s search experience. This is currently not the case as parent documents are unaware of their child documents beyond the indexing process.

I’m now thoroughly enjoying tackling these issues and working on the next iteration of this RecursiveMergeExistingDoc processor!

Update – 06/02/15

The source code is now available on github

• This has been posted earlier on lab.klemespen.com.
• Even though some of these ideas are not what you’d normally implement in a business environment, some of the concepts can obviously be transferred over to businesses trying to provide an efficient workplace for its employees.
• Norch is developed by Fergus McDowall, an employee of Comerio.

What if you could index your whole life and make this lifeindex available through search? What would that look like, and how could it help you? Refinding information is obviously one of the use case for this type of search. I’m guessing there’s a lot more, and I’m curious to figure them out.

Actions and reactions instead of web pages

I had the lifeindex idea for a little while now. Originally the idea was to index everything I browsed. From what I know and where Norch is, it would take a while before I was anywhere close to achieving that goal. Then I thought of IFTTT, and saw it as a ‘next best thing’. But then it hit me that now I’m indexing actions, and that’s way better than pages. But what I’m missing from most sources now are the reactions to my actions. If I have a question, I also want to crawl and index the answer. If I have a statement, I want to get the critique indexed.

IFTTT and similar services (like Zapier) is quite limiting in their choice of triggers. Not sure if this is because of choices done by those services or limitations from the sites they crawl/pull information from.

A quick fix for this, and a generally good idea for Search Engines, would be to switch from a preview of your content to the actual content in the form of an embed-view. Here exemplified:

Will embed-view of your content replace the preview-pane in modern #search #engine solutions? Why preview when you can have the real deal?

— Espen Klem (@eklem) November 24, 2014

Technology: Hello IFTTT, Google SpreadSheet and Norch

IFTTT is triggered by my actions, and stores some data to a series of spreadsheets on Google Drive. These spreadsheets can deliver JSON. After a little document processing these JSON-files can be fed to the Norch-indexer.

Why hasn’t this idea popped up earlier?

Search engines used to be hardware guzzling technology. With Norch, the “NOde seaRCH” engine, that has changed. Elasticsearch and Solr are easy and small compared to i.e. SharePoint Search, but still it needs a lot of hardware. Norch can run on a Raspberry Pi, and soon it will be able to run in your browser. Maybe data sets closer to small data is more interesting than big data?

Norch running on a Raspberry Pi

Why using a search engine?

It’s cheap and quick. I’m not a developer, and I’ll still be able to glue all these sources together. Search engines are often a good choice when you have multiple sources. IFTTT and Google SpreadSheet makes it even easier, normalising the input and delivering it as JSON.

How far in the process have I come?

So far, I’ve set up a lot of triggers/sources at IFTTT.com:

• Instagram: When posting or liking both photos and videos.
• Flickr: When posting an image, creating a set or linking a photo.
• Google Calendar: When adding something to one of my calendars.
• Facebook: When i post a link, is tagged, post a status message.
• Twitter: When I tweet, retweet, reply or if somebody mentions me.
• Youtube: When I post or like a video.
• GitHub: When I create an issue, gets assigned to an issue or any issues that I part take in is closed.
• WordPress: When new posts or comments on posts.
• Android location tracking: When I enter and exit certain areas.
• Android phone log: Placed, received and missed calls.
• Gmail: Starred emails.

And gotten a good chunk of data. Indexing my SMS’es felt a bit creepy, so I stopped doing that. And storing email just sounded too excessive, but I think starred emails would suit the purpose of the project.

Those Google Drive documents are giving me JSON. Not JSON that I can feed directly Norch-indexer, it needs a little trimming.

Issues discovered so far

Manual work

This search solution needs a lot of manual setup. Every trigger needs to be set up manually. Everytime a new trigger is triggered, I get a new spreadsheet that needs a title row added. Or else, the JSON variables will look funny, since first row is used for variable names.

The spreadsheets only accepts 2000 rows. After that a new file is created. Either I need to delete content, rename the file or reconfigure some stuff.

Level of maturity

IFTTT is a really nice service, and they treat their users well. But, for now, it’s not something you can trust fully.

Cleaning up duplicates and obsolete stuff

I have no way of removing stuff from the index automatically at this point. If I delete something I’ve added/written/created, it will not be reflected in the index.

Missing sources

Books I buy, music I listen to, movies and TV-series I watch. Or Amazon, Spotify, Netflix and HBO. Apart from that, there are no Norwegian services available through IFTTT.

History

The crawling is triggered by my actions. That leaves me without history. So, i.e. new contacts on LinkedIn is meaningless when I don’t get to index the existing ones.

Next steps

JSON clean-up

I need to make a document processing step. Norch-document-processor would be nice if it had handled JSON in addition to HTML. Not yet, but maybe in the future? Anyway, there’s just a small amount of JSON clean-up before I got my data in and index.

When this step is done, a first version can be demoed.

UX and front-end code

To show the full potential, I need some interaction design of the idea. For now they’re all in my head. And these sketches needs to be converted to HTML, CSS and Angular view.

Embed codes

Figure out how to embed Instagram, Flickr, Facebook and LinkedIn-posts, Google Maps, federated phonebook search etc.

OAUTH configuration

Set up OAUTH NPM package to access non-public spreadsheets on Google Drive. Then I can add some of the less open information I have stored.

Part two is now available here!

At the beginning of this year Christopher Vig wrote a great post about indexing an SQL database to the internet’s current search engine du jour, Elasticsearch. This first post in a two part series will show that Apache Solr is a robust and versatile alternative that makes indexing an SQL database just as easy. The second will go deeper into how to make leverage Solr’s features to create a great backend for a people search solution.

Solr ships with a configuration driven contrib called the DataImportHandler. It provides a way to index structured data into Solr in both full and incremental delta imports. We will cover a simple use case of the tool i.e. indexing a database containing personnel data to form the basis of a people search solution. You can also easily extend the DataImportHandler tool via various APIs to pre-process data and handle more complex use cases.

For now, let’s stick with basic indexing of an SQL database.

Setting up our environment

Before we get started, there are a few requirements:

1. Java 1.7 or greater
2. For this demo we’ll be using a MySQL database
3. A copy of the sample employees database
4. The MySQL jdbc driver

With that out of the way, let’s get Solr up and running and ready for database indexing:

1. Download Solr and extract it to a directory of your choice.
2. Open solr-4.9.0/example/solr/collection1/conf/solrconfig.xml in a text editor and add the following within the config tags: 
3. In the same directory, open schema.xml and add this this line  
4. Create a lib subdir in solr-4.9.0/solr/collection1/ and extract the MySQL jdbc driver jar into it. It’s the file called mysql-connector-java-{version}-bin.jar
5. To start Solr, open a terminal and navigate to the example subdir in your extracted Solr directory and run java -jar start.jar

When started this way, Solr runs by default on port 8983. If you need to change this, edit solr-4.9.0/example/etc/jetty.xml and restart Solr.

Navigate to http://localhost:8983/solr and you should see the Solr admin GUI splash page. From here, use the Core Selector dropdown button to select the default core and then click on the Dataimport option. Expanding the Configuration section should show an XML response with a stacktrace with a message along the lines of Can't find resource 'db-data-config.xml' in classpath. This is normal as we haven’t actually created this file yet, which stores the configs for connecting to our target database.

We’ll come back to that file later but let’s make our demo database now. If you haven’t already downloaded the sample employees database and installed MySQL, now would be a good time!

Setting up our database

Assuming your MySQL server is installed and running, access the MySQL terminal and create the empty employees database: create database employees;

Exit the MySQL terminal and import the employees.sql into your empty database, ensuring that you carry out the following command from the same directory as the employees.sql file itself: mysql -u root -p employees < employees.sql

You can test this was successful by logging into the MySql server and querying the database, as shown here on the right.

Having successfully created and populated your employee database, we can now create that missing db-data-config.xml file.

Indexing our database

In your Solr conf directory, which contains the schema.xml and solrconfig.xml we previously modified, create a new file called db-data-config.xml.

Its contents should look like the example below. Make sure to replace the user and password values with yours and feel free to modify or remove the limit parameter. There’s approximately 30’000 entries in the employees table in total

We’re now going to make use of Solr’s REST-like HTTP API with a couple of commands worth saving. I prefer to use the Postman app on Chrome and have created a public collection of HTTP requests, which you can import into Postman’s Collections view using this url: https://www.getpostman.com/collections/9e95b8130556209ed643

For those of you not using Chrome, here are the commands you will need: First let’s reload the core so that Solr is
aware of the new db-data-config.xml file we have created.
Next, we index our database with the HTTP request or from within the Solr Admin GUI on the DataImport page.

Here we have carried out a full index of our database using the full-import command parameter. To only retrieve changes since the last import, we would use delta-import instead.

We can confirm that our database import was successful by querying our index with the “Retrieve all” and “Georgi query” requests.

Finally, to schedule reindexing you can use a simple cronjob. This one, for example, will run everyday at 23:00 and retrieve all changes since the previous indexing operation:

Conclusion

So far we have successfully

• Setup a database with content
• Indexed the database into our Solr index
• Setup basic scheduled delta reindexing

In the next part of this two part series we will look at how to process our indexed data. Specifically, with a view to making a good people search solution. We will implement several features such as phonetic search, spellcheck and basic query completion. In the meantime, let’s carry on the conversation in the comments below!

While both platforms are based on Lucene, Elasticsearch has been built for scaling from day one, while it is often said Solr added this as an afterthought. Developers also generally find it very easy to interact with since it uses a JSON-based API model. Much could be said about the differences between the two platforms, but without going into a long list of technical details, Elasticsearch felt like a breath of fresh air when we first laid eyes on it a few years ago.

Elasticsearch or Solr? Google trends will give you a nice input to the dilemma.

Today the company behind the product, announced they have raised a whopping 70M USD in funding, bringing the total up to over 100M USD in the last 18 months!

For everyone loving the product, and everyone considering what platform to go with for their enterprise search needs, this is excellent news! Even though it is an open source product, and everyone in theory can submit suggested changes, bugfixes etc – there’s definite value in having a group of full time, dedicated developers building the product.

This means the product and the ecosystem around it will continue to evolve, improve and increase for a long time with considerable force.

So – for all those wondering what horse to put their bets on for future open source development;  Picking the fastest running horse is often better than picking the horse that’s been around the longest.

Comperio is an official partner with Elasticsearch.

Please join us at our breakfast seminar in Oslo in a few days (June 11). Elasticsearch will be there to present alongside Microsoft and Google.

Getting the correct result at the top of your search results isn’t easy. Anyone working within search quickly realizes this. Tuning the underlying ranking model is a job that just doesn’t end. There is an entire profession about search engine optimization, making sure your site gets as high as possible on Google (and Bing, I guess). If it is not the top result on Google, it is somehow your fault and not Google’s.

This year’s summer internship program has been centered around the concept “Search driven websites”. Over the course of the summer we’ve gone from a concept and a vision to a prototype website, and we’d like to share the potential we’ve uncovered along the way with you.

What is a search driven website?

On a search driven website, everything is driven by search. More useful and less painfully obvious, a search driven website will get all its content, all its navigation and menus as a result of a search query. This allows a different and more dynamic approach to the structuring and placement of content, in contrast to a typical folder based, editor dependent website where someone has to decide exactly which articles and images will reside on the front page, or any page, at any one time.

A traditional site requires the user to spend time learning and navigating a cramped and complex menu structure to find the content desired. We believe that navigation can and should be done in a better way, through the use of an ever-ready search bar, dynamic navigators and relevant links. Even when displaying a single page, a search is performed to retrieve that article along with related results.

There are immediate advantages to using search as the main logic module of a website. Take the front page: Set the site’s opening search to “*” – search everything – and make the search engine order the results by relevance, popularity and date. Display the results as proper, readable articles, and you get a site containing the content you want to display without having to manually specify what goes where. A clear advantage here is that popular results will be displayed automatically on the front page, letting many of your readers find what they want immediately. This will also let your site adapt to your readers’ interests automatically, and not just when you perform a user test or go through a slew of analytics.

A unified experience

Within this vision of search driven websites is the idea that just as the front page content is retrieved with a search, the reader’s personal searches should also be displayed like the front page. Let’s use a specific example. Below is a screen shot from the front page of aftenposten.no, a large Norwegian news site.

This looks pretty good. Now let’s search for China and see how it looks.

Not looking so good anymore. When we perform a search, it no longer looks like a newspaper. It no longer IS a newspaper, it is a mini Google. What if, instead, the results looked just like the front page?

Now the search becomes an integrated part of the website. Readers no longer have to enter “search mode” to find what they are looking for, they simply type something into the search bar and get a site specifically tailored to their interests. It’s clean, fresh and feels natural. This part of the search driven vision could easily be implemented in most websites out there, and we believe that unifying the search and the rest of the website would make the Internet a more comfortable place to be.

Reduce maintenance and integration costs

The big advantage of search is that you can simply tell the search engine what you want and it will find the best content for you. We know this might sound obvious, but think about the consequences. Life becomes easier when all you need to do is ask the search to fill your site with content. Remember, both the site administrator and the readers can have a say in what the search engine will look for. To base your website on search can be a good alternative when it is important that relevant and updated content is displayed at all times. If your content is located in many different places and you have different readers with different information needs, a search based solution could be your best bet.

With a search based website, the decisions become all about what to search for in which situations. By gathering statistics and usage patterns, the search can be tweaked to give each reader a personalized experience. Whenever you want to add a new page to your site about a particular topic, you’d simply get the content of your new page through a customized search. The rest is just creating relevant content.

When businesses get  new software systems, merge with or acquire other businesses, you often get several sources for your content. This is an organizational pain and comes with high software integration costs. In the purest version of the search driven vision, all you would need to do is make sure your search engine can index the new source. There is no need to move your content or cross/double post articles, as long as the content is in the search engine it’ll be available through your search driven site.

Readers often want an overview of what exists on your site as they look for content. Search based navigation could be the answer. Up until recent years, people have been used to navigating through mutually exclusive folders, and this is the way many websites are built today. An alternative approach is to let the search engine figure out which categories exist, for example by observing where pages have been posted or what they are tagged with. Let clicks in the menu navigation be searches for the menu keywords, or a search for elements with a specific category tag. The real locations and structure of your content will no longer be important. This reduces the integration work on site navigation by not having to consider the actual locations of the content, just make sure the new content is categorized or tagged properly.

 Our prototype

This summer we have created a prototype for this concept, giving the rest of Comperio an example implementation and a platform for further discussions. You can  see our prototype at the address below.  To check out the category navigation click the “show categories” in the bottom right corner.

beta.comperio.no

The site is composed of four parts. A search engine, an admin interface, a search API and a client. The client is created using Javascript and handlebars.js, the search engine is an Apache Solr instance and the admin interface and search API are both contained in a wordpress plugin. The client requests content from the search API, which uses the search engine as a back end to find relevant content before it is returned to the client. Everything found in the prototype is completely search driven, including the front page, single page view, related items and category navigation. The admin interface lets you control indexing as well as different search parameters.

A last note

Over the summer we have built a website completely based on search. We believe that using search as the main content delivery method has great potential. Businesses today have vast amounts of information, and there is an ever increasing need for sharing, accessibility and flexibility. These are challenges that can’t be ignored and we have enjoyed uncovering, discussing, and hopefully getting closer to a solution to these problems in light of the concept of search driven web sites.