Ansible is a free software platform for configuring and managing computers, and I’ve been using it a lot lately to manage the ELK stack. Elasticsearch, Logstash and Kibana.

I can define a list of servers I want to manage in a YAML config file – the so called inventory:

[elasticearch-master]
es-master1.mydomain.com
es-master2.mydomain.com
es-master3.mydomain.com

[elasticsearch-data]
elk-data1.mydomain.com
elk-data2.mydomain.com
elk-data3.mydomain.com

[kibana]
kibana.mydomain.com

And define the roles for the servers in another YAML config file – the so called playbook:

- hosts: elasticsearch-master
  roles:
    - ansible-elasticsearch

- hosts: elasticsearch-data
  roles:
    - ansible-elasticsearch

- hosts: logstash
  roles:
    - ansible-logstash

- hosts: kibana
  roles:
    - ansible-kibana

Each group of servers may have their own files containing configuration variables.

elasticsearch_version: 2.1.0
elasticsearch_node_master: false
elasticsearch_heap_size: 1000G

Ansible is used for configuring the ELK stack vagrant box at https://github.com/comperiosearch/vagrant-elk-box-ansible, which was recently upgraded with Elasticsearch 2.1, Kibana 4.3 and Logstash 2.1

The same set of Ansible roles can be applied when the configuration needs to move into production, by applying another set of variable files with modified host names, certificates and such. The possible ways to do this are several.

How does it work?

Ansible is agent-less. This means, you do not install anything (an agent) on the machines you control. Ansible needs only to be installed on the controlling machine (Linux/OSX) and  connects to the managed machines (some support for windows, even) using SSH. The only requirement on the managed machines is python.

Happy ansibling!

In this blog post we introduce a Vagrant box to easily create configurable and reproducible development environments for ELK (Elasticsearch, Logastash and Kibana). At Comperio, we mainly use this box for query log analysis using the ELK stack.
In case you don’t know, Vagrant is a free and open-source software that combines VirtualBox (a virtualization software) with configuration management softwares such as Puppet and Chef.

ELK stack up and running in two commands

$ git clone https://github.com/comperiosearch/vagrant-elk-box.git
$ vagrant up

By cloning this github repo and then typing “vagrant up”, you will be installing elasticsearch, logstash, kibana and nginx (the latter used to serve kibana).

Elasticsearch will be running on port 9200, as usual, which is forwarded to the host machine. As for Kibana, it will be served on port 5601 (also accessible from the host OS).

How does it work?
As mentioned above, Vagrant is a wrapper around VirtualBox and some configuration management software. In our box, we use pure shell scripting and Puppet to configure the ELK stack.
There are two essential configuration files in this box: Vagrantfile and the Puppet manifest default.pp.
Vagrantfile includes the settings of the virtual box such as operating system, memory size, number of CPUs, forwarded ports, etc…

Vagrantfile also includes a shell script that installs, among other things, the official Puppet modules for elasticsearch and logstash. By using that shell script we stay away from git submodules which were used in another Vagrant image we made earlier for elasticsearch.

In the Puppet manifest, default.pp, we define what version of elasticsearch to install and make sure that it is running as a service.

We do the same for logstash and additionally link the default logstash configuration file to this file under /Vagrant/confs/logstash which is shared with the host OS. Finally, we install nginx and Kibana, and configure Kibana to run on port 5601 (by linking the nginx conf file to this file in the Vagrant directory also).