Elasticsearch is an open source search and analytics engine that extends the limits of full-text search through a robust set of APIs and DSLs, to deliver a flexible and almost limitless search experience.

L is for Logstash

One of the most popular open source log parser solutions on the market, Logstash has the possibility of reading any data source and extracting the data in JSON format, easy to use and running in minutes.

K is for Kibana

A data visualization engine, Kibana allows the user to create custom dashboards and to analyze Elasticsearch data on-the-fly and in real-time.

Getting set up

To start using this technology, you just need to install the three above mentioned components, which actually means downloading and unzipping three archive files.

The data flow is this: the log files are text files residing in a folder. Logstash will use a configuration file to read from the logs and parse all the entries. The parsed data will be sent to Elasticsearch for storing. Once here, it can be easily read and displayed by Kibana.

Parsing SharePoint ULS log files with Logstash

We will now focus on the most simple and straightforward way of getting this to work, without any additional configuration or settings. Our goal is to open Kibana and be able to configure some charts that will help us visualize and explore what type of entries we have in the SharePoint ULS logs, and to be able to search the logs for interesting entries.

To begin, we need some ULS log files from SharePoint that will be placed in a folder on the server (I am working on a Windows Server virtual environment) where we are testing the ELK stack. My ULS logs are located here: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\LOGS

As an example, the first line in one of my log files looks like this:

05/06/2014 10:20:20.85 wsstracing.exe (0x0900)                 0x0928SharePoint Foundation         Tracing Controller Service    5152InformationTracing Service started.

The next step is to build the configuration file. This is a text file with a .config extension, located by default in the Logstash folder. The starting point for the content of this file would be: 

input {  
 file {  
  type => "sharepointlog" 
    path => ["[folder where the logs reside]/*.log"] 
   start_position => "beginning" 
   codec => "plain" 
} 
} 
filter  
{ 
 } 
output  
{    
 elasticsearch {  
embedded => true 
 } 
}

The Input defines the location of the logs and some reading parameters, like the starting position where Logstash will begin parsing the files. The Output section defines the location of the parsed data, in our case the Elasticsearch instance installed on the same server.

Now for the important part, the Filter section. The Filter section contains one or more GROK patterns that are used by Logstash for identifying the format of the log entries. There are many types of entries, but we are focusing on the event type and message, so we have to parse all the parameters up to the message part in order to get what we need.

The documentation is pretty detailed when it comes to GROK and a pattern debugger website with a GROK testing engine is available online, so you can develop and test your patterns before actually running them in Logstash.

So this is what I came up with for the SharePoint ULS logs:

filter { 
   if [type] == "sharepointlog" { 
grok { 
match => [ "message",  
"(?<parsedtime>%{MONTHNUM}/%{MONTHDAY}/%{YEAR} %{HOUR}:%{MINUTE}:%{SECOND}) \t%{DATA:process} \(%{DATA:processcode}\)(\s*)\t%{DATA:tid}(\s*)\t(?<area>.*)(\s*)\t(?<category>.*)(\s*)\t%{WORD:eventID}(\s*)\t%{WORD:level}(\s*)\t%{DATA:eventmessage}\t%{UUID:CorrelationID}"] 
match => [ "message",  
"(?<parsedtime>%{MONTHNUM}/%{MONTHDAY}/%{YEAR} %{HOUR}:%{MINUTE}:%{SECOND}) \t%{DATA:process} \(%{DATA:processcode}\)(\s*)\t%{DATA:tid}(\s*)\t(?<area>.*)(\s*)\t(?<category>.*)(\s*)\t%{WORD:eventID}(\s*)\t%{WORD:level}(\s*)\t%{DATA:eventmessage}"] 
match => [ "message",  
“(?<parsedtime>%{MONTHNUM}/%{MONTHDAY}/%{YEAR} %{HOUR}:%{MINUTE}:%{SECOND})%{GREEDYDATA}\t%{DATA:process} \(%{DATA:processcode}\)(\s*)\t%{DATA:tid}(\s*)\t(?<area>.*)(\s*)\t(?<category>.*)(\s*)\t%{WORD:eventID}(\s*)\t%{WORD:level}(\s*)\t%{DATA:eventmessage}"] 
} 
date { 
match => ["parsedtime","MM/dd/YYYY HH:mm:ss.SSS"] 
} 
   } 
}

Logstash in action

All that’s left to do is to get Logstash going and see what comes out. Run the following on the command line:

logstash.bat agent -f "sharepoint.conf"

This runs logstash as an agent, so it will monitor the file or the folder you specify in the input section of the config for changes. If you are indexing a folder where files appear periodically, you don’t need to worry about restarting the process, it will continue on its own.

Kibana time

Now let’s create a new dashboard in Kibana and see what was indexed. The most straight-forward panel type is Histogram. Make no changes to the default settings of this panel (Chart value = count, Time field = @timestamp) and you should see something similar to this:

To get some more relevant information, we can add some pie charts and let them display other properties that we have mapped, for example ‘process’ or ‘area’.  

Now let’s turn this up a notch: through Kibana we can take a look at the errors in the SharePoint logs. Create a pie chart that displays the “level” field. By clicking on the “Unexpected” slice in this chart you will filter all the dashboard on this value.  

Kibana will automatically refresh the page, the filter itself will be displayed in the “Filter” row and all you will see are the “Unexpected” events.  Time to turn to the help of a Table chart: by displaying the columns you select on the Fields section of this chart, you can view and sort the log entries for a more detailed analysis of the unexpected events.

As the Logstash process runs as an agent, you can monitor the SharePoint events in real-time! So there you have it, SharePoint log analysis using ELK.

But very often a small aspect of search is neglected: The zero result page, or the zero results strategy (There are many reasons for this, I’ll explain one later*). Yesterday I came across a search that I think could benefit greatly from zero result strategy. I was reading an article about some really good Norwegian authors, and decided to buy some of the books at a Norwegian E-book store called ebok.no.  The E-book market in Norway is not very big, even if the general book market is. Few books are published as E-books and you often have to wait a while before a book is available as an E-book. I was lucky and found two out of the three books I wanted. The last one: “Bergeners” by Tomas Espedal I misspelled a couple of times before I got it right. Even then I had to check a regular bookstore to see if it was actually correctly spelled.

Solution: Monetize search

What if ebok.no, every time a user got a zero result, did a lookup in a database to check if the search query matched an actual book. If yes, say the following:

Sorry, we don’t have the book “[title]” by [author] available as an E-book.

Would you like us to send an email when/if it is available as an E-book at ebok.no?

It the user agrees, you have achieved one and maybe two things. Firstly, you’ve made your users do a one time subscription to pay you money even if you slapped them with a zero result. Secondly, if the user wasn’t a registered user already, you may have convinced him or her to become a registered user later on because of good customer service.

*One reason a zero result strategy often is neglected is that you need the search to be in production for a while before you have enough data to analyse your zero results. In most cases, by that time, the project has ended.