Elasticsearch is an open source search and analytics engine that extends the limits of full-text search through a robust set of APIs and DSLs, to deliver a flexible and almost limitless search experience.

L is for Logstash

One of the most popular open source log parser solutions on the market, Logstash has the possibility of reading any data source and extracting the data in JSON format, easy to use and running in minutes.

K is for Kibana

A data visualization engine, Kibana allows the user to create custom dashboards and to analyze Elasticsearch data on-the-fly and in real-time.

Getting set up

To start using this technology, you just need to install the three above mentioned components, which actually means downloading and unzipping three archive files.

The data flow is this: the log files are text files residing in a folder. Logstash will use a configuration file to read from the logs and parse all the entries. The parsed data will be sent to Elasticsearch for storing. Once here, it can be easily read and displayed by Kibana.

Parsing SharePoint ULS log files with Logstash

We will now focus on the most simple and straightforward way of getting this to work, without any additional configuration or settings. Our goal is to open Kibana and be able to configure some charts that will help us visualize and explore what type of entries we have in the SharePoint ULS logs, and to be able to search the logs for interesting entries.

To begin, we need some ULS log files from SharePoint that will be placed in a folder on the server (I am working on a Windows Server virtual environment) where we are testing the ELK stack. My ULS logs are located here: C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\LOGS

As an example, the first line in one of my log files looks like this:

05/06/2014 10:20:20.85 wsstracing.exe (0x0900)                 0x0928SharePoint Foundation         Tracing Controller Service    5152InformationTracing Service started.

The next step is to build the configuration file. This is a text file with a .config extension, located by default in the Logstash folder. The starting point for the content of this file would be: 

input {  
 file {  
  type => "sharepointlog" 
    path => ["[folder where the logs reside]/*.log"] 
   start_position => "beginning" 
   codec => "plain" 
} 
} 
filter  
{ 
 } 
output  
{    
 elasticsearch {  
embedded => true 
 } 
}

The Input defines the location of the logs and some reading parameters, like the starting position where Logstash will begin parsing the files. The Output section defines the location of the parsed data, in our case the Elasticsearch instance installed on the same server.

Now for the important part, the Filter section. The Filter section contains one or more GROK patterns that are used by Logstash for identifying the format of the log entries. There are many types of entries, but we are focusing on the event type and message, so we have to parse all the parameters up to the message part in order to get what we need.

The documentation is pretty detailed when it comes to GROK and a pattern debugger website with a GROK testing engine is available online, so you can develop and test your patterns before actually running them in Logstash.

So this is what I came up with for the SharePoint ULS logs:

filter { 
   if [type] == "sharepointlog" { 
grok { 
match => [ "message",  
"(?<parsedtime>%{MONTHNUM}/%{MONTHDAY}/%{YEAR} %{HOUR}:%{MINUTE}:%{SECOND}) \t%{DATA:process} \(%{DATA:processcode}\)(\s*)\t%{DATA:tid}(\s*)\t(?<area>.*)(\s*)\t(?<category>.*)(\s*)\t%{WORD:eventID}(\s*)\t%{WORD:level}(\s*)\t%{DATA:eventmessage}\t%{UUID:CorrelationID}"] 
match => [ "message",  
"(?<parsedtime>%{MONTHNUM}/%{MONTHDAY}/%{YEAR} %{HOUR}:%{MINUTE}:%{SECOND}) \t%{DATA:process} \(%{DATA:processcode}\)(\s*)\t%{DATA:tid}(\s*)\t(?<area>.*)(\s*)\t(?<category>.*)(\s*)\t%{WORD:eventID}(\s*)\t%{WORD:level}(\s*)\t%{DATA:eventmessage}"] 
match => [ "message",  
“(?<parsedtime>%{MONTHNUM}/%{MONTHDAY}/%{YEAR} %{HOUR}:%{MINUTE}:%{SECOND})%{GREEDYDATA}\t%{DATA:process} \(%{DATA:processcode}\)(\s*)\t%{DATA:tid}(\s*)\t(?<area>.*)(\s*)\t(?<category>.*)(\s*)\t%{WORD:eventID}(\s*)\t%{WORD:level}(\s*)\t%{DATA:eventmessage}"] 
} 
date { 
match => ["parsedtime","MM/dd/YYYY HH:mm:ss.SSS"] 
} 
   } 
}

Logstash in action

All that’s left to do is to get Logstash going and see what comes out. Run the following on the command line:

logstash.bat agent -f "sharepoint.conf"

This runs logstash as an agent, so it will monitor the file or the folder you specify in the input section of the config for changes. If you are indexing a folder where files appear periodically, you don’t need to worry about restarting the process, it will continue on its own.

Kibana time

Now let’s create a new dashboard in Kibana and see what was indexed. The most straight-forward panel type is Histogram. Make no changes to the default settings of this panel (Chart value = count, Time field = @timestamp) and you should see something similar to this:

To get some more relevant information, we can add some pie charts and let them display other properties that we have mapped, for example ‘process’ or ‘area’.  

Now let’s turn this up a notch: through Kibana we can take a look at the errors in the SharePoint logs. Create a pie chart that displays the “level” field. By clicking on the “Unexpected” slice in this chart you will filter all the dashboard on this value.  

Kibana will automatically refresh the page, the filter itself will be displayed in the “Filter” row and all you will see are the “Unexpected” events.  Time to turn to the help of a Table chart: by displaying the columns you select on the Fields section of this chart, you can view and sort the log entries for a more detailed analysis of the unexpected events.

As the Logstash process runs as an agent, you can monitor the SharePoint events in real-time! So there you have it, SharePoint log analysis using ELK.

The display templates are located in the site collection root site, under the _catalogs/Display Templates folder. Each template has two files: the html file and a javascript file. The javascript file is the one that SharePoint uses, while the html exists in order to make it easier for us developers to create and customize the display templates. Once an html file has been modified, SharePoint generates the corresponding updated javascript file, so the process is fully automated.

So what are the steps involved in creating a new display template?

In SharePoint designer, after opening the root website, navigate to the Display Templates folder. Here, in the Search subfolder, you will find the display templates that SharePoint uses for many types of results – documents, web sites, people and others. Choose a display template as a starting point for your own, depending on the result type that you are targeting, copy it and then rename it. Once saved, you will see that SharePoint has already generated the corresponding javascript file. Modify the html file as you wish. What is left to do now is to link the template to an actual result type, so that SharePoint knows what to associate your display template to. This can be done through the interface, by navigating to Site Settings – Search Result Types.

Let’s take it step by step.

Duplicating an existing display template

Open the site collection with SharePoint designer. Navigate to the Display Templates folder under _catalogs\masterpage\display templates. Here you will find some Folders that SharePoint uses to group Display Templates together.

The Word item template is in the Search folder. Make a copy of the Item_Word.html and rename it to ComperioItem_Word.html

Edit the file and inside you will see some html and javascript code.

Edit the title field so that your custom template has a different display name than the original one.

The <mso:CustomDocumentProperties> tag contains some properties of the template. Some important properties are:

• TargetControlType – tells SharePoint what the template is used for (search results, web parts, filters)
• ManagedPropertyMapping – contains a list of the managed properties that are or can be used by the current template

The <body> tag contains the rendering logic for the display template.

The template can only recognize the managed properties that are listed in the header. To use another managed property, add it here. We are using the Created property that displays the document created date.

<mso:ManagedPropertyMapping msdt:dt="string"> 
'Title':'Title','Path':'Path', [………..] , ‘Created’:’Created’ 
</mso:ManagedPropertyMapping>

Now that the template is aware of the property, you can use it by adding it to the body. Add the line below in the Body of the rendering template:

<div class="ms-descriptiontext"> _#= ctx.CurrentItem.Created =#_ </div>

Mapping the display template to a result type

Now we are going to manually link the display template to a result type, so that we can see that it works.

In order for SharePoint to use your custom template, it needs to know of it. So navigate to Site Actions – Site Settings and under Site Collection Administration choose Search Result Types. Create a New Result Type for your template like below:

• Choose a Name
• Choose Microsoft Word as content to match
• Choose your custom display template

Now go to the results page and search for a word document. The results should display a date looking like this:

We have only been scratching the surface and a lot more customization can be done here, for example formatting the date value in javascript by using a function like format(“yyyy-MM-dd”) on a Date object. Combined, SharePoint and javascript have (almost) endless possibilities.

Happy coding!