Tuning Graylog, Elasticsearch, and MongoDB for optimized cluster performance This has been an article a long time in the making. One problem with making changes to a complex clustered environment is that you may have to wait long periods of time to gather data that either shows an improvement or shows a negative impact. Some other considerations just make total sense, if you can afford them. Running on SSDs is going to perform far better than spinning disks.

Setup HA Cluster Services on CentOS 7 Install HA Cluster components Install pacemaker and the cluster control software on both nodes that will be part of the cluster(corosync is pulled in as a dependency) yum install pacemaker pcs Enable and start the cluster management service systemctl enable pcsd.service systemctl start pcsd.service Enable corosync and pacemaker to start on boot on all nodes systemctl enable corosync.service systemctl enable pacemaker.service Set the hacluster user’s password

Setup for Logging Setup rsyslog aggregator nodes (Optional) Setup Note: As part of the overall design, an HA cluster allows aggregating logs to the Central Log Repository with as little loss of logs as possible due to downtime or maintenance. Below are steps for building an HA cluster and setting up rsyslog for CENTOS 7. Install/upgrade to the latest rsyslog yum update rsyslog Create an rsyslog spool directory (this will be needed later)

Setup for Logging Setup rsyslog node Install/upgrade to the latest rsyslog yum update rsyslog Create an rsyslog spool directory (this will be needed later) mkdir /var/lib/rsyslog Setup Note: A custom rsyslog.conf is available for the CLR node that allows receiving logs on tcp port 514 by default. Copy the content from the appendixes into the appropriate files. Path: /etc/rsyslog.conf rsyslog.conf for CLR server Edit the rsyslog config vi /etc/rsyslog.conf Uncomment the lines for the action and change server.

Additional Setup for master node Setup Graylog Web UI on master node Setup Note: newer versions of graylog do not require a separate install for the web interface anymore so we can make a few firewall rule changes and be good. Configure Graylog WebUI firewalld rules Let ’s make some firewall rule changes specifically to allow web traffic. If for some reason you aren’t using a firewall then you can skip this.

Additional setup for master node Setup graylog-server on master node Install instructions from http://docs.graylog.org/en/2.2/pages/installation.html Setup Note: This deployment is not using a prebuilt rpm package, many of the next steps will be moving files, creating directories, creating additional files, and setting up the proper permissions on the linux command line. An rpm package is available but because when this guide for first written, the RPM only had support for openjdk v1.

Additional Setup for master node Install mongodb on master node Install instructions from https://docs.mongodb.com/manual/administration/install-on-linux/ Create repo file for mongodb vi /etc/yum.repos.d/MongoDB-3.4.repo Insert this text [mongodb-org-3.4] name=MongoDB Repository baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.4/x86_64/ gpgcheck=1 enabled=1 gpgkey=https://www.mongodb.org/static/pgp/server-3.4.asc</pre> Install mongodb yum install mongodb-org Set mongod to start on boot systemctl enable mongod .service Allow mongodb to use port 27017 for communication semanage port -a -t mongod_port_t -p tcp 27017 Start mongodb systemctl start mongod Setup Note: By default, mongodb does not enforce any kind of user authentication.

Setup Elasticsearch cluster nodes Install Elasticsearch In this example we are building out a three node cluster but this can scale up to fit whatever cluster size you choose. View Elasticsearch setup and configuration docs https://www.elastic.co/guide/en/elasticsearch/reference/2.4/index.html Install Java yum install java-1.8.0-openjdk-headless.x86_64 Import signing key from elastic.co rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch Create repo file vi /etc/yum.repos.d/Elasticsearch.repo Insert this text [elasticsearch-2.x] name=Elasticsearch repository for 2.x packages baseurl=http://packages.elastic.co/elasticsearch/2.x/centos gpgcheck=1 gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch enabled=1 Install elasticsearch yum install elasticsearch

Partitioning layout Setup Note: The partition layout for elasticsearch nodes may vary depending on how much data storage is needed. It makes sense to create mount points with the idea they may grow in the future. Worksheet for estimating proper sizing that can be helpful https://community.graylog.org/t/sizing-estimator/644 For ES data nodes, ES master node, and rsyslog storage node /dev/sda1 mount on /boot 500MB vg_local-lv_swap swap partition 1024MB (up to twice size of installed RAM) vg_local-lv_root mount on / minimum 10240MB (10GB) vg_local-lv_home mount on /home minimum 15360MB (15GB), expandable vg_local-lv_tmp mount on /tmp minimum 1024MB (1GB) vg_local-lv_usr mount on /usr minimum 4096MB (4GB) vg_local-lv_var mount on /var minimum 8000MB (8GB), expandable For rsyslog aggregator nodes (Optional) /dev/sda1 mount on /boot 500MB vg_local-lv_swap 1024MB up to twice size of installed RAM vg_local-lv_root mount on / minimum of 8192MB vg_local-lv_home mount on /home minimum of 10240MB up to whatever size vg_local-lv_tmp mount on /tmp minimum of 1024MB vg_local-lv_usr mount on /usr minimum of 4096MB vg_local-lv_var mount on /var minimum of 8000MB up to whatever size For OSSEC node (Optional) /dev/sda1 mount on /boot 500MB vg_local-lv_swap 1024MB up to twice size of installed RAM vg_local-lv_root mount on / minimum of 8192MB vg_local-lv_home mount on /home minimum of 10240MB up to whatever size vg_local-lv_tmp mount on /tmp minimum of 1024MB vg_local-lv_usr mount on /usr minimum of 4096MB vg_local-lv_var mount on /var minimum of 8000MB up to whatever size Install prerequisite apps Setup Note: Some services should be available on the network such as a timeserver and DNS services otherwise, some assumptions made throughout this document will not be correct.

System Build Overview The next steps are to build the environment; starting with the elasticsearch (ES) nodes and the log parser/search frontend because they require certain components to be identical. The process assumes the use of a minimal OS install using CENTOS 7 but any major NIX based OS can be used (just remember the commands might be different). Start by building three servers; two will be ES data nodes and one will be used as the ES master node.