Elasticsearch will by default allow anyone access to all data. The Shield plugin allows locking down Elasticsearch using authentication from the internal esusers realm, Active Directory (AD)  or LDAP . Using AD, you can map groups defined in your Windows domain to roles in Elasticsearch. For instance, you can allow people in the Fishery department access only to  fish-indexes, and give complete control to anyone in the IT department.

To use Shield in production, you have to buy an Elasticsearch subscription, however, you get a 30-day trial when installing the license manager. So let’s hurry up and see how this works out in Kibana.

In this post, we will install Shield and connect to Active Directory (AD) for authentication. After having made sure we can authenticate with AD, we will add SSL encryption everywhere possible. We will add authentication for the Kibana server using the built in authentication realm esusers, and if time allows at the end, we will create two user groups, each with access to its own index, and check how it all looks when accessed in Kibana 4.

Prerequisites

You will need a previously installed Elasticsearch and Kibana. The most recent versions should work, I have used Elasticsearch 1.7 and Kibana 4.1.1  If you need a machine to test on, I can personally recommend the vagrant-elk-box you can find here: The following guide assumes the file locations of the vagrant-elk-box, if you install differently, you will probably know where to look. Ask an adult for help.

For Active Directory, you need to be on a domain that uses Active Directory. That would probably mean some kind of Windows work environment.

Installing Shield

If you’re on the vagrant box you should begin the lesson by entering the vagrant box using the commands

vagrant up
vagrant ssh

Install the license manager

 sudo /usr/share/elasticsearch/bin/plugin -i elasticsearch/license/latest

Install Shield

 sudo /usr/share/elasticsearch/bin/plugin -i elasticsearch/shield/latest

Restart elasticsearch. (service elasticsearch restart)

Check out the logs,  you should find some information regarding when your Shield license will expire (logfile location:  /var/log/elasticsearch/vagrant-es.log)

Integrating Active Directory

The next step involves figuring out a thing or two about your Active Directory configuration. First of all you need to know the address. Now you need to be on  your windows machine, open cmd.exe and type

set LOGONSERVER

The name of your AD should pop back.  Add a section similar to the following into the elasticsearch.yml file (at /etc/elasticsearch/elasticsearch.yml)

shield.authc.realms:
  active_directory:
    type: active_directory
    domain_name: superdomain.com
    unmapped_groups_as_roles: true
    url: ldap://ad.superdomain.com

Type in the address to your AD in the url: field (where it says url: ldap://ad.superdomain.com). If your logonserver is ad.cnn.com, you should type in url: ldap://ad.cnn.com

Also, you need to figure out your domain name and type it in correctly.

NB: Be careful with the indenting! Elasticsesarch cares a lot about correct indenting, and may even refuse to start without telling you why if you make a mistake.

Finding the Correct name for the Active Directory group

Next step involves figuring out the name for the Group you wish to grant access to. You may have called your group “Fishermen”, but that is probably not exactly what it’s called in AD.

Microsoft has a very simple and nice tool called Active Directory Explorer . Open the tool and enter the adress you just found from the LOGONSERVER (remember? it’s only 10 lines above)

You may have to click and explore a little to find the groups you want. Once you find it, you need the value for the “distinguishedName” attribute. You can double click on it and copy out from the “Object”.

This is an example from my AD

CN=Rolle IT,OU=Groups,OU=Oslo,OU=Comperiosearch,DC=comperiosearch,DC=com

Now this value represents a group which we want to map to a role in elasticsearch.

Open the file /etc/elasticsearch/shield/role-mapping.yml. It should look similar to this

# Role mapping configuration file which has elasticsearch roles as keys
# that map to one or more user or group distinguished names

#roleA:   this is an elasticsearch role
#  - groupA-DN  this is a group distinguished name
#  - groupB-DN
#  - user1-DN   this is the full user distinguished name
power_user:
  - "CN=Rolle IT,OU=Groups,OU=Oslo,OU=Comperiosearch,DC=comperiosearch,DC=com"
#user:
# - "cn=admins,dc=example,dc=com" 
# - "cn=John Doe,cn=other users,dc=example,dc=com"

I have uncommented the line with “power_user:” and added a line below containing the distinguishedName from above.

By restarting elasticsearch, anyone in the “Rolle IT” group should now be able to log in (and nobody else (yet)).

To test it out, open http://localhost:9200 in your browser. You should be presented with a login box where you can type in your username/password. In case of failure, check out the elasticsearch logs (at /var/log/elasticsearch/vagrant-es.log).

If you were able to log in, that means Active Directory authentication works. Congratulations!  You deserve a refreshment. Some strong coffee, will go down well with the next sections, where we add encrypted communications everywhere we can.

SSL  - Elasticsearch

Authentication and encrypted communication go hand in hand. Without SSL, username and password is transferred in plaintext on the wire. For this demo we will use self-signed certificates. Keytool comes with Java, and is used to handle certificates for Elasticsearch.  The following command will generate a self-signed certficate and put it in a JKS file named self-signed.jks. (swap out  $password with your preferred password)

keytool -genkey -keyalg RSA -alias selfsigned -keystore self-signed.jks -keypass $password -storepass $password -validity 360 -keysize 2048 -dname "CN=localhost, OU=orgUnit, O=org, L=city, S=state, C=NO"

Copy the certificate into /etc/elasticsearch/

Modify  /etc/elasticsearch/elasticsearch.yml by adding the following lines:

shield.ssl.keystore.path: /etc/elasticsearch/self-signed.jks
shield.ssl.keystore.password: $password
shield.ssl.hostname_verification: false
shield.transport.ssl: true
shield.http.ssl: true

(use the same password as you used when creating the self-signed certificate )

Restart Elasticsearch again, and watch the logs for failures.

Try to open https://localhost:9200 in your browser (NB: httpS not http)

https://localhost:9200

You should a screen warning you that something is wrong with the connection. This is a good sign! It means your certificate is actually working! For production use you could use your own CA or buy a proper certificate, which both will avoid the ugly warning screen.

SSL – Active directory

Our current method of connecting to Active Directory is unencrypted – we need to enable SSL for the AD connections.

1. Fetch the certificate from your Active Directory server (replace ldap.example.com with the LOGONSERVER from above)

echo | openssl s_client -connect ldap.example.com:6362>/dev/null| openssl x509 > ldap.crt

2. Import the certificate into your keystore (located at /etc/elasticsearch/)

keytool -import-keystore self-signed.jks -file ldap.crt

3. Modify AD url in elasticsearch.yml
change the line

url: ldap://ad.superdomain.com

to

url: ldaps://ad.superdomain.com

Restart elasticsearch and check logs for failures

Kibana authentication with esusers

With Elasticsearch locked down by Shield, it means no services can search or post data either. Including Kibana and Logstash.

Active Directory is great, but I’m not sure I want to use it for letting the Kibana server talk to Elasticsearch. We can use the Shield built in user management system, esusers. Elasticsearch comes with a set of predefined roles, including roles for Logstash, Kibana4 server and Kibana4 user. (/etc/elasticsearch/shield/role-mapping.yml on the vagrant-elk box if you’re still on that one).

Add a new kibana4_server user, granting it the role kibana4_server, using this command:

cd /usr/share/elasticsearch/bin/shield  
./esusers useradd kibana4_server -p secret -r kibana4_server

Adding esusers realm

The esusers realm is the default one, and does not need to be configured if that’s the only realm you use. Now since we added the Active Directory realm we must add another section to the elasticsearch.yml file from above.

It should end up looking like this

shield.authc.realms:
  esusers:
    type: esusers
    order: 0
  active_directory:
    order: 1
    type: active_directory
    domain_name: superdomain.com
    unmapped_groups_as_roles: true
    url: ldap://ad.superdomain.com

The order parameter defines in what order elasticsearch should try the various authentication mechanisms.

Allowing Kibana to access Elasticsearch

Kibana must be informed of the new user we just created. You will find the kibana configuration file at /opt/kibana/config/kibana.yml.

Add in the username and password you just created. You also need to change the address for elasticsearch to using https

# The Elasticsearch instance to use for all your queries.
elasticsearch_url: "https://localhost:9200"

# If your Elasticsearch is protected with basic auth, this is the user credentials
# used by the Kibana server to perform maintence on the kibana_index at statup. Your Kibana
# users will still need to authenticate with Elasticsearch (which is proxied thorugh
# the Kibana server)
kibana_elasticsearch_username: kibana4_server
kibana_elasticsearch_password: secret

Restart kibana and elasticsearch, and watch the logs for any errors. Try opening Kibana at  http://localhost:5601, type in your login and password. Provided you’re in the group you gave access earlier, you should be able to login.

Creating SSL for Kibana

Once you have enabled authorization for Elasticsearch, you really need to set SSL certificates for Kibana as well. This is also configured in kibana.yml

verify_ssl: false
# SSL for outgoing requests from the Kibana Server (PEM formatted)
ssl_key_file: "kibana_ssl_key_file"
ssl_cert_file: "kibana_ssl_cert_file"

You can create a self-signed key and cert file for kibana using the following command:

openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -days 365 -nodes

Configuring AD groups for Kibana access

Unfortunately, this part of the post is going to be very sketchy, as we are desperately running out of time. This blog is much too long already.

Elasticsearch already comes with a list of predefined roles, among which you can find the kibana4 role.  The kibana4 role allows read/write access to the .kibana index, in addition to search and read access to all indexes. We want to limit access to just one index for each AD group. The fishery group shall only access the fishery index, and the finance group shall only acess the finance index. We can create roles that limit access to one index by copying the kibana4 role, giving it an appropriate name and changing the index:’*’ section to map to only the preferred index.

The final step involves mapping the Elasticsearch role into an AD role. This is done in the role_mapping.yml file, as mentioned above.

Only joking of course, that wasn’t the last step. The last step is restarting Elasticsearch, and checking the logs for failures as you try to log in.

Securing Elasticsearch

Shield brings enterprise authentication to Elasticsearch. You can easily manage access to various parts of  Elasticsearch management and data by using Active Directory groups.

This has been a short dive into the possibilities, make sure to contact Comperio if you should need  help in creating a solution with Elasticsearch and Shield.

Keynote

Fighting the crowds to find a seat for the keynote at Day 2 at elastic{ON}15 we were blocked by a USB stick with the curious caption  Microsoft (heart) Linux. Things have certainly changed.

Microsoft

The keynote, led by Elastic SVP of sales Aaron Katz, included Pablo Castro of Microsoft who was keen to explain how this probably isn’t so far from the truth. Elasticsearch is used  internally in several Microsoft products among Linux and other open source software and this is a huge change from the Microsoft we know from around five years ago. Pablo revealed some details towards how elasticsearch is used as data storage and search platform in MSN, Microsoft Dynamics and Azure Search. Microsoft truly has gone through some fundamental changes lately embracing open source both internally and externally. We see this as a demonstration of the power of open source and the huge value of Elastic(search) brings to  many organizations. As Jordan Sissel said in the keynote yesterday “If a user has a problem, it is a bug”. This is a philosophical stance towards a conception of software as an enabler of  creativity and growth, in contrast to viewing software as a fixed product packaged for sale.

Goldman Sachs

Microsofts contribution was in the middle part of the keynote. The first part was a discussion with Don Duet, managing director of Goldman Sachs. Goldman Sachs provides financial services on a global scale, and has been on the forefront of technology since its inception in 1869. They were an early adopter of Elasticsearch since it was as an easy to use search and analytics tool for big data. Goldman Sachs is now using elasticsearch extensively as a key part of their technological stack.

NASA

The most mind blowing part of the keynote was the last one held by two chaps from the Jet Propulsion Labs team at NASA, Ricky Ma and Don Isla. They first showed their awesome internal search with previews, and built in rank tuning. Then they talked about the Mars Curiosity rover, a robot planted on Mars which runs around taking samples and selfies. It constantly sends data back to earth where the JPL team analyzes the operations of the rover. Elasticsearch is naturally at the center of this interplanetary operation, nothing less.

Mars Curiosity Rover Selfie

The remainder of the day contained sessions across the same three tracks as the first day. In addition five tracks of birds of a feather or “lounge” sessions were held where people gathered in smaller groups to discuss various topics.  Needless to say the breadth of the program meant we were stretched thin. We chose to focus on three topics that are of particular importance to our customers: aggregations, security & Shield, and resiliency

More aggregations

Adrien Grand & Colin Goodheart-Smithe did a deep dive into the details of aggregations and how they are computed. In particular how to tune them and the results in terms of execution complexity. A key point is the approximations that are employed to compute some of the aggregations which involve certain trade offs in speed over accuracy. Aggregations are a very powerful feature requiring some some planning to be feasible and efficient.

Security/Shield

Uri Boness talked about Shield and the current state of authentication & authorization, He provided some pointers to what is on the roadmap for the coming releases. Unfortunately, there does not appear to be any concrete plans for providing built in document level security. This is a sought after feature that would certainly make the product more interesting in many enterprise settings. Then again, there are companies who provide connector frameworks that include security solutions for elasticsearch. We had a chat with some of them at the conference, including Enonic, SearchBlox and Search Technologies.

Facebook

Peter Vulgaris from Facebook explained how they are using elasticsearch. To me, the story resembled Microsoft’s. Facebook has heaps of data, and lots of use cases for it. Once they started to use elasticsearch it was widely adopted in the company and the amount of data indexed grew ever larger which forced them to think more closely about how they manage their clusters.

Resiliency

Elasticsearch is a distributed system, and as such shares the same potential issues as other distributed systems. Boaz Leskes & Igor Motov explained the measures that have been undertaken in order to avoid problems such as “split-brain” syndrome. This is when a cluster is confused about what node should be considered the master. Data safety and security are important features of Elasticsearch and there is a continuous effort in place in these areas.

Lucene

Stepping back to day 1 and the Lucene session featuring the mighty Robert Muir, we learned that Lucene version 5 includes a lot of improvements. Especially performance wise regarding compression both on indexing and query times which enables faster execution times and reduces resource consumption. There has also been made efforts to the Lucene core enabling a merging of query and filter as two sides of the same coin. After all a query is just  a filter with a relevance score. On another note Lucene will now handle caching of queries by itself.

Wrapping it up

Elastic{ON}15 stands as a confirmation of the attitude that were essential in the creation of the elasticsearch project. The visions that guided the early development are still valid today, except the scale is larger. The recent emphasis on stability, security and resiliency will welcome a new wave of users and developers.

At the same time there is a continuous exploration and development into big data related analytics but with the speed and agility we have come to expect from Elasticsearch.

Thanks for this year, looking forwards to next!

The default installation optionally creates and configures a self-signed CA, and issues all necessary certificates. This is handy for development and test installations, when it might not be possible to use a PKI infrastructure. Or simply just when you don’t want to flesh out the money for a trusted root CA.

However, you don’t want to have your production environment running on self-signed certificates. You want to use a proper, trusted, CA.

If you opt to use a trusted CA right from the get-go, when you first install FSIS, the process for doing so is fairly well-documented. But if you want to migrate an existing installation, going from using the self-signed certificates to relying on a trusted CA, there’s at least one caveat you should know about. In the documentation about renewing certificates, and changing the CA, there’s a small note saying:

The procedure assumes that you do not change the name of the CA. SecurityConfig -useNewCerts offers no way to update the CA subject name in the configuration. The name of the CA is used for authorization throughout the system. If it changes, all configuration files referencing this name on all hosts must be updated and all the hosts must be restarted.

This is crucial information, because if you update the CA, it’s very likely that the subject name of the new certificate does not match the old from the self-signed version. Other than that, the process of moving to a trusted CA is straight-forward. Put the new certificates in place, run SecurityConfig with the -useNewCerts options, and restart the FSIS service.

This is just what I did yesterday. Unfortunately for me, after deploying the new certificates, my client application got an Access Denied response, or at best a “The credentials supplied to the package were not recognized”. On my FSIS node, I set up a trace of the communication, but that only told me that the connection was dropped. After lots of debugging, I luckily stumbled over the note in the documentation.

So I needed to update those “configuration files”. As the author wasn’t kind enough to specify which these were, I ran a string search over the whole FSIS install directory. The files that I needed to update were the Node.config files in FSIS’s AdminNode and ImsNode services:

C:\Users\[My FSIS user]\AppData\Local\FSIS\Nodes\Fsis\AdminNode1\Node.config
C:\Users\[My FSIS user]\AppData\Local\FSIS\Nodes\Fsis\InteractionEngineNode1\Node.config

Depending on your setup, you might have several such files, spread over several machines. In any case, those files contain the following XML node:

<add key="IssuerAuthorizationManager.AuthorizedIssuer" value="CN=fsisIssuer@MY_HOSTNAME" />

After updating the CN name, and restarting the FSIS service, my client application finally started to work as expected.

UPDATE

The procedure above; updating the InteractionEngineNodeX\Node.config, will of course only affect the settings for that particular node. After some more digging around, I’ve found out how to make the changes kick in also for new nodes added later on.

The template file for the IMS Node.config files lives here (yes, no file suffix):

%FSIS_HOME%\Resources\Templates\InteractionEngineNode

This file contains the following line:

<add key="IssuerAuthorizationManager.AuthorizedIssuer" value="${Managed.CertificateIssuerName}" />

You can replace ${Managed.CertificateIssuerName} with something like CN=fsisIssuer@MY_HOSTNAME, but even better is to replace the actual variable being expanded. It is defined under the <appsettings> here:

%FSIS_HOME%\Host Controller\HostControllerService.exe.config