Elasticsearch – Just Another IT Guy

After you have set up your ELK Stack and have been using it for a while (see Step-By-Step Install ELK Stack), a question should start creeping into your head; How do I stop my Elasticsearch Indexes from growing endlessly? Now you could occasionally go into Kibana and delete the indexes via the GUI found there, but we’re sysadmins! We automate things! Luckily Elastic has provided a utility for managing indexes named Curator, which is easily ran as a cron job. Win! Be sure to visit the Elastic Curator page and get an idea of what you can do with it, and roughly how it is configured. We are going to configure it to delete all indexes beginning with winlogbeat- and filebeat- that are older than 90 days in this example, so let’s get to setting that up. I will be showing you the most recent version as of writing this, Curator version 5.5.4.

Installing & Configuring Curator

1. Start by downloading the DEB package which is hidden on the APT install page. I usually place these in /tmp for easy cleanup.

wget https://packages.elastic.co/curator/5/debian9/pool/main/e/elasticsearch-curator/elasticsearch-curator_5.5.4_amd64.deb

1	wget https://packages.elastic.co/curator/5/debian9/pool/main/e/elasticsearch-curator/elasticsearch-curator_5.5.4_amd64.deb

2. Once you have that downloaded, let’s install it.

sudo dpkg -i elasticsearch-curator_5.5.4_amd64.deb

1	sudo dpkg -i elasticsearch-curator_5.5.4_amd64.deb

3. Curator will now be installed at /opt/elasticsearch-curator, but we don’t want to mess with anything in there. I created a hidden directory in my home directory named curator as the documentation suggests as default, and then created a configuration file in that directory for Curator.

sudo mkdir /home/username/.curator

sudo vi /home/username/.curator/curator.yml

sudo mkdir /home/username/.curator

sudo vi /home/username/.curator/curator.yml

I placed the following configuration in the curator.yml file.

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
client:
  hosts:
    - 127.0.0.1
  port: 9200
  url_prefix:
  use_ssl: False
  certificate:
  client_cert:
  client_key:
  ssl_no_validate: False
  http_auth:
  timeout: 30
  master_only: False

logging:
  loglevel: INFO
  logfile: '/home/username/.curator/log.log'
  logformat: default
  blacklist: ['elasticsearch', 'urllib3']

---

# Remember, leave a key empty if there is no value. None will be a string,

# not a Python "NoneType"

client:

hosts:

- 127.0.0.1

port: 9200

url_prefix:

use_ssl: False

certificate:

client_cert:

client_key:

ssl_no_validate: False

http_auth:

timeout: 30

master_only: False

logging:

loglevel: INFO

logfile: '/home/username/.curator/log.log'

logformat: default

blacklist: ['elasticsearch', 'urllib3']

This is pretty straightforward. It tells Curator to connect to Elasticsearch found on localhost (127.0.0.1) on port 9200 without SSL and send basic log info to a file at /home/username/.curator/log.log.

4. Now we need to tell Curator what to do now that it knows what to connect to and what not. We do this with an action file, which I creatively named action.yml.

sudo vi /home/username/.curator/action.yml

1	sudo vi /home/username/.curator/action.yml

I placed the following configuration in the action.yml based off of the example found here in the documentation.

---
# Remember, leave a key empty if there is no value.  None will be a string,
# not a Python "NoneType"
#
# Also remember that all examples have 'disable_action' set to True.  If you
# want to use this action as a template, be sure to set this to False after
# copying it.
actions:
  1:
    action: delete_indices
    description: >-
      Delete indices older than 90 days (based on index name), for winlogbeat-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: winlogbeat-
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 90
  2:
    action: delete_indices
    description: >-
      Delete indices older than 90 days (based on index name), for filebeat-
      prefixed indices. Ignore the error if the filter does not result in an
      actionable list of indices (ignore_empty_list) and exit cleanly.
    options:
      ignore_empty_list: True
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: filebeat-
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 90

---

# Remember, leave a key empty if there is no value. None will be a string,

# not a Python "NoneType"

# Also remember that all examples have 'disable_action' set to True. If you

# want to use this action as a template, be sure to set this to False after

# copying it.

actions:

action: delete_indices

description: >-

Delete indices older than 90 days (based on index name), for winlogbeat-

prefixed indices. Ignore the error if the filter does not result in an

actionable list of indices (ignore_empty_list) and exit cleanly.

options:

ignore_empty_list: True

disable_action: False

filters:

- filtertype: pattern

kind: prefix

value: winlogbeat-

- filtertype: age

source: name

direction: older

timestring: '%Y.%m.%d'

unit: days

unit_count: 90

action: delete_indices

description: >-

Delete indices older than 90 days (based on index name), for filebeat-

prefixed indices. Ignore the error if the filter does not result in an

actionable list of indices (ignore_empty_list) and exit cleanly.

options:

ignore_empty_list: True

disable_action: False

filters:

- filtertype: pattern

kind: prefix

value: filebeat-

- filtertype: age

source: name

direction: older

timestring: '%Y.%m.%d'

unit: days

unit_count: 90

5. Now we can test it with the –dry-run argument. This will log what Curator WOULD do if it were not being run with the –dry-run argument, it does not actually perform the actions.

sudo curator --config /home/username/.curator/curator.yml --dry-run /home/username/.curator/action.yml

1	sudo curator --config /home/username/.curator/curator.yml --dry-run /home/username/.curator/action.yml

Check the log to see all indexes older than 90 days would have been deleted.

vi /home/username/.curator/log.log

1	vi /home/username/.curator/log.log

6. Assuming that all checks out, we just have to make a cron job to run this thing for us. I like to run it daily, but it’s dealers choice. Start by making your script in the appropriate directory.

sudo vi /etc/cron.daily/curator.sh

1	sudo vi /etc/cron.daily/curator.sh

And added the following:

curator --config /home/username/.curator/curator.yml /home/username/.curator/action.yml

1	curator --config /home/username/.curator/curator.yml /home/username/.curator/action.yml

7. Now we just need to add the actual cronjob.

sudo crontab -e

1	sudo crontab -e

Append the following to the bottom of the file:

0 6 * * * /etc/cron.daily/curator.sh

1	0 6 * * * /etc/cron.daily/curator.sh

This configures the cronjob to run the curator.sh script that we made in step 6 to run every day at 6am as root. Curator is now set up to manage your indexes!

Elasticsearch, Logstash, and Kibana (aka ELK Stack) are very powerful tools for storing, analyzing, and visualizing log data in a centralized location. That being said, it can be quite the headache to actually get up and running if it is your first experience with it. Having spent the time pouring over the documentation provided by Elastic, which I must say is quite impressive, and struggling through getting the ELK stack up and running I figured I would make a step-by-step. Some things to note before we get started:

This will be for the current most recent version 6.3.2 released on July 24, 2018.
I will set up an nginx reverse proxy to access Kibana.
I will not be including SSL setup.
I will be installing all components of the stack with DEB packages.
Java 8 is required prior to ELK setup. See my post Install Java JDK/JRE on Ubuntu Server without APT.
- Java 10 is not compatible with the Logstash 6.3.2. I learned this the hard way, take my word for it.
I will be showing BEATS configurations in a separate post. This will be only the ELK stack setup.

I am not using APT repositories for anything because I have been burned by the upgrade process in the past with ELK, so I just manually upgrade as necessary. Now, let’s get this thing started.

Install & Configure Elasticsearch

1. Start by navigating to the Elastic downloads page and select Elasticsearch.

2. Login to your Ubuntu box and download the DEB package. I put it in tmp for easy cleanup.

cd /tmp
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.2.deb

1 2	cd /tmp wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.2.deb

3. Now download the checksum and compare against your downloaded package. It should return OK.

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.2.deb.sha512
shasum -a 512 -c elasticsearch-6.3.2.deb.sha512

1 2	wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.2.deb.sha512 shasum -a 512 -c elasticsearch-6.3.2.deb.sha512

4. Install Elasticsearch.

sudo dpkg -i elasticsearch-6.3.2.deb

1	sudo dpkg -i elasticsearch-6.3.2.deb

5. Open the Elasticsearch config file found at /etc/elasticsearch/elasticsearch.yml and uncomment/edit the following settings.

custer.name: my-cluster-name

node.name: my-node-name

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

network.host: localhost

custer.name: my-cluster-name

node.name: my-node-name

path.data: /var/lib/elasticsearch

path.logs: /var/log/elasticsearch

network.host: localhost

This will configure your cluster name as my-cluster-name, your node (or server) as my-node-name, your data storage location as /var/lib/elasticsearch, your log location as /var/log/elasticsearch, and your host as localhost (127.0.0.1). These are pretty default settings and I don’t see many reasons to change them, but do so if you wish.

6. Restart/Reload the service, daemon, and enable the service.

sudo systemctl restart elasticsearch.service

sudo systemctl daemon-reload

sudo systemctl enable elasticsearch.service

sudo systemctl restart elasticsearch.service

sudo systemctl daemon-reload

sudo systemctl enable elasticsearch.service

7. Test Elasticsearch.

curl http://localhost:9200

1	curl http://localhost:9200

Which should return:

{
  "name" : "my-node-name",
  "cluster_name" : "my-cluster-name",
  "cluster_uuid" : "ltd7ik4TQ1-BfAJ-VieBZw",
  "version" : {
    "number" : "6.3.2",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "053779d",
    "build_date" : "2018-07-20T05:20:23.451332Z",
    "build_snapshot" : false,
    "lucene_version" : "7.3.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

{

"name" : "my-node-name",

"cluster_name" : "my-cluster-name",

"cluster_uuid" : "ltd7ik4TQ1-BfAJ-VieBZw",

"version" : {

"number" : "6.3.2",

"build_flavor" : "default",

"build_type" : "deb",

"build_hash" : "053779d",

"build_date" : "2018-07-20T05:20:23.451332Z",

"build_snapshot" : false,

"lucene_version" : "7.3.1",

"minimum_wire_compatibility_version" : "5.6.0",

"minimum_index_compatibility_version" : "5.0.0"

"tagline" : "You Know, for Search"

}

Install & Configure Kibana

1. Download Kibana DEB package from Elastic downloads page.

wget https://artifacts.elastic.co/downloads/kibana/kibana-6.3.2-amd64.deb

1	wget https://artifacts.elastic.co/downloads/kibana/kibana-6.3.2-amd64.deb

2. Install Kibana.

sudo dpkg -i kibana-6.2.3-amd64.deb

1	sudo dpkg -i kibana-6.2.3-amd64.deb

3. Open the Kibana config file found at /etc/kibana/kibana.yml and uncomment the following:

server.host: "localhost"

1	server.host: "localhost"

4. Restart/Reload daemon, enable and start the service.

sudo systemctl daemon-reload

sudo systemctl enable kibana.service

sudo systemctl start kibana.service

sudo systemctl daemon-reload

sudo systemctl enable kibana.service

sudo systemctl start kibana.service

Install & Configure Nginx (Source)

1. Install nginx.

sudo apt update

sudo apt install -y nginx

sudo apt update

sudo apt install -y nginx

2. Setup user for basic authentication.

sudo -v

echo "kibanaadmin:`openssl passwd -apr1`" | sudo tee -a /etc/nginx/htpasswd.users

sudo -v

echo "kibanaadmin:`openssl passwd -apr1`" | sudo tee -a /etc/nginx/htpasswd.users

Enter a password for the user when prompted.

3. Configure nginx by clearing /etc/nginx/sites-available/default and inputting the following:

server {
    listen 80;

    server_name example.com;

    auth_basic "Restricted Access";
    auth_basic_user_file /etc/nginx/htpasswd.users;

    location / {
        proxy_pass http://localhost:5601;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_cache_bypass $http_upgrade;        
    }
}

server {

listen 80;

server_name example.com;

auth_basic "Restricted Access";

auth_basic_user_file /etc/nginx/htpasswd.users;

location / {

proxy_pass http://localhost:5601;

proxy_http_version 1.1;

proxy_set_header Upgrade $http_upgrade;

proxy_set_header Connection 'upgrade';

proxy_set_header Host $host;

proxy_cache_bypass $http_upgrade;

}

This will configure nginx as a reverse-proxy for Kibana, while also requiring the username and password set up in step two.

4. Test nginx configuration and restart service.

sudo nginx -t

sudo systemctl restart nginx.service

sudo nginx -t

sudo systemctl restart nginx.service

Install & Configure Logstash

1. Download the Logstash DEB package from the Elastic downloads page.

wget https://artifacts.elastic.co/downloads/logstash/logstash-6.3.2.deb

1	wget https://artifacts.elastic.co/downloads/logstash/logstash-6.3.2.deb

2. Install Logstash.

sudo dpkg -i logstash-6.3.2.deb

1	sudo dpkg -i logstash-6.3.2.deb

3. Create the file /etc/logstash/conf.d/10-beats.conf and input the following:

input {
  beats {
    port => 5044
    ssl => false
  }
}

input {

beats {

port => 5044

ssl => false

}

This will configure logstash to listen for beats applications on port 5044 without requiring SSL.

4. Create the file /etc/logstash/conf.d/50-output.conf and input the following:

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
  }
}

output {

elasticsearch {

hosts => ["localhost:9200"]

sniffing => true

manage_template => false

index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"

}

This will configure logstash to output beats data to elasticsearch on this host to index which named is determined by specified variables. In this case, the beats application name – date. Ex. winlogbeat-2018.08.23.

5. Test your Logstash configuration.

sudo /usr/share/logstash/bin/logstash -t -f /etc/logstash/conf.d/

1	sudo /usr/share/logstash/bin/logstash -t -f /etc/logstash/conf.d/

6. Restart and enable Logstash service.

sudo systemctl restart logstash.service

sudo systemctl enable logstash.service

sudo systemctl restart logstash.service

sudo systemctl enable logstash.service

At this point you should now have a functional ELK server that will accept input from BEATS!

This Week In Enterprise Tech

TWiET 438: AI Ethics & Bias - Are you introducing bias into your AI projects? April 10, 2021

Discord, Slack, and Malware API Copying is Fair Use, rules Supreme Court Blanketing the planet in broadband Fake Netflix Android App FlixOnline hijacks WhatsApp sessions Hospital Pricing data and online search results Triveni Gandhi on Artificial Intelligence Ethics and Data Bias Hosts: Louis Maresca, Brian Chee, and Curt Franklin Guest: Triveni Gandhi Download or subscribe […]

TWiT

TWiET 437: IT Business Playground Boundaries - Rackspace: Are we too often chasing the new shiny technology? April 3, 2021

Universities nationwide targeted by hackers due to security flaws in a file-transfer product AR indoor navigation and other new features coming to Google Maps Attackers know enterprises have a problem with overprivileged users 10Mbps upload speeds are good enough, according to AT&T Adam Selipsky to succeed Andy Jassy as AWS CEO Ubiquiti's "Catastrophic" breach Jeff […]

TWiT

TWiET 436: Identity Mode: Random - Chip shortage affects automakers, 60TB hard drives might be coming, Sonrai Security on identity sprawl in the post SolarWinds world March 27, 2021

Open SSL fixes flaw that hackers can use to crash servers Automakers extending production cuts due to chip shortage Where Rust will fit into Linux according to Linus Torvalds iOS and Android hackers shut down by Facebook Almost 50% of popular Android apps are built on open source components with high-risk vulnerability An executive order […]

TWiT

S	M	T	W	T	F	S
				1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28	29	30

The Mad Ramblings of an Abnormal SysAdmin

Tag: Elasticsearch

Install & Configure Elastic Curator for Index Management

Installing & Configuring Curator

Step-By-Step Install ELK Stack on Ubuntu 18.04

Install & Configure Elasticsearch

Install & Configure Kibana

Install & Configure Nginx (Source)

Install & Configure Logstash