After you have set up your ELK Stack and have been using it for a while (see Step-By-Step Install ELK Stack), a question should start creeping into your head; How do I stop my Elasticsearch Indexes from growing endlessly? Now you could occasionally go into Kibana and delete the indexes via the GUI found there, but we’re sysadmins! We automate things! Luckily Elastic has provided a utility for managing indexes named Curator, which is easily ran as a cron job. Win! Be sure to visit the Elastic Curator page and get an idea of what you can do with it, and roughly how it is configured. We are going to configure it to delete all indexes beginning with winlogbeat- and filebeat- that are older than 90 days in this example, so let’s get to setting that up. I will be showing you the most recent version as of writing this, Curator version 5.5.4.
Installing & Configuring Curator
1. Start by downloading the DEB package which is hidden on the APT install page. I usually place these in /tmp for easy cleanup.
1 |
wget https://packages.elastic.co/curator/5/debian9/pool/main/e/elasticsearch-curator/elasticsearch-curator_5.5.4_amd64.deb |
2. Once you have that downloaded, let’s install it.
1 |
sudo dpkg -i elasticsearch-curator_5.5.4_amd64.deb |
3. Curator will now be installed at /opt/elasticsearch-curator, but we don’t want to mess with anything in there. I created a hidden directory in my home directory named curator as the documentation suggests as default, and then created a configuration file in that directory for Curator.
1 2 3 |
sudo mkdir /home/username/.curator sudo vi /home/username/.curator/curator.yml |
I placed the following configuration in the curator.yml file.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
--- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" client: hosts: - 127.0.0.1 port: 9200 url_prefix: use_ssl: False certificate: client_cert: client_key: ssl_no_validate: False http_auth: timeout: 30 master_only: False logging: loglevel: INFO logfile: '/home/username/.curator/log.log' logformat: default blacklist: ['elasticsearch', 'urllib3'] |
This is pretty straightforward. It tells Curator to connect to Elasticsearch found on localhost (127.0.0.1) on port 9200 without SSL and send basic log info to a file at /home/username/.curator/log.log.
4. Now we need to tell Curator what to do now that it knows what to connect to and what not. We do this with an action file, which I creatively named action.yml.
1 |
sudo vi /home/username/.curator/action.yml |
I placed the following configuration in the action.yml based off of the example found here in the documentation.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 |
--- # Remember, leave a key empty if there is no value. None will be a string, # not a Python "NoneType" # # Also remember that all examples have 'disable_action' set to True. If you # want to use this action as a template, be sure to set this to False after # copying it. actions: 1: action: delete_indices description: >- Delete indices older than 90 days (based on index name), for winlogbeat- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly. options: ignore_empty_list: True disable_action: False filters: - filtertype: pattern kind: prefix value: winlogbeat- - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 90 2: action: delete_indices description: >- Delete indices older than 90 days (based on index name), for filebeat- prefixed indices. Ignore the error if the filter does not result in an actionable list of indices (ignore_empty_list) and exit cleanly. options: ignore_empty_list: True disable_action: False filters: - filtertype: pattern kind: prefix value: filebeat- - filtertype: age source: name direction: older timestring: '%Y.%m.%d' unit: days unit_count: 90 |
5. Now we can test it with the –dry-run argument. This will log what Curator WOULD do if it were not being run with the –dry-run argument, it does not actually perform the actions.
1 |
sudo curator --config /home/username/.curator/curator.yml --dry-run /home/username/.curator/action.yml |
Check the log to see all indexes older than 90 days would have been deleted.
1 |
vi /home/username/.curator/log.log |
6. Assuming that all checks out, we just have to make a cron job to run this thing for us. I like to run it daily, but it’s dealers choice. Start by making your script in the appropriate directory.
1 |
sudo vi /etc/cron.daily/curator.sh |
And added the following:
1 |
curator --config /home/username/.curator/curator.yml /home/username/.curator/action.yml |
7. Now we just need to add the actual cronjob.
1 |
sudo crontab -e |
Append the following to the bottom of the file:
1 |
0 6 * * * /etc/cron.daily/curator.sh |
This configures the cronjob to run the curator.sh script that we made in step 6 to run every day at 6am as root. Curator is now set up to manage your indexes!