Elasticstack (ELK) and pfSense Firewall – Monitoring system performance with Elastic Metricbeats

Elasticstack (ELK) and pfSense Firewall – Monitoring system performance with Elastic Metricbeats

Introduction

This is an unexpected article that came about while reviewing the documentation for Elasticbeats (As used for the Elasticstack with pfSense and Suricata series). It covers the implementation on pfSense of Metricbeat, another element of the Elasticbeats package, a tool used for shipping system performance and utilization metrics to Elasticsearch for reporting and monitoring.

Warning: Implementation requires adding additional packages and kernel modules to the standard pfSense installation. This has risks and is likely unsupported by pfSense. Additionally, the addition of *any* additional software on a security device has trade-offs with the potential to increase any attack surface and introduce another path for exploits. You should understand the potential consequences of any such changes.

Installation of Metricbeat

Elasticbeats package

The installation of the Elasticbeats package, which contains all of the beats shippers, is covered in Elasticstack (ELK), Suricata and pfSense Firewall Part 1: Elasticbeats and pfSense configuration  go and review that article and come back here once you have the beats package installed.

Metricbeat Prerequisites

Metricbeat utilises portions of the freeBSD Linux(r) Binary Compatibilty. This is not available in the pfSense customisation of FreeBSD and so we need to reinstate this in order to allow Metricbeat to access process information.

The instructions below were created after review of the Metricbeat FAQ and the ever-helpful pfSense community forum. Specifically with reference to the following pages:

The specific instructions for installing the items required for Metricbeat on pfSense are as follows:

Determine the freeBSD version that your pfSense is running:

/root: uname -r
10.3-RELEASE-p19

Proceed to the freeBSD site and download the Installer/liveCD image that corresponds with the pfSense version. In my instance it was 10.3.

Extract the following files. I used 7-Zip to open the ISO image directly and copy out the files. You could boot the liveCD on a virtual machine or install the full freeBSD version to access the same.

/boot/kernel/linux_common.ko
/boot/kernel/linprocfs.ko

Upload these files to your pfSense instance. I used SCP to upload the files to pfSense.

Ensure they are placed in the /boot/kernel directory and have the same file security as other .ko files in the location. Use chmod to alter if required:

chmod 555 /boot/kernel/linux_common.ko
chmod 555 /boot/kernel/linprocfs.ko

Load the Linux Proc Filesystem Kernel Module and check the Linux compatibility modules are loaded:

/root: kldload linprocfs.ko

/root: kldstat

Id Refs Address Size Name 
1 18 0xffffffff80200000 225f380 kernel 
2 1 0xffffffff82611000 29eb coretemp.ko 
3 1 0xffffffff82614000 d299 dummynet.ko 
4 1 0xffffffff82622000 9faf linprocfs.ko 
5 1 0xffffffff8262c000 683b linux_common.ko 

Create the directory to be used as the Linux Proc Filesystem mount-point:

/root: mkdir -p /compat/linux/proc

Mount the linproc File system:

/root: mount -t linprocfs /dev/null /compat/linux/proc

Confirm information is available:

/root: ls /compat/linux/proc/

0 13 18 284 322 4 56221 58529 69697 72363 cmdline meminfo scsi uptime
1 14 19 28481 324 42931 56937 6 69866 73333 cpuinfo mounts self version
10 15 2 3 326 5 56973 65594 7 76872 devices mtab stat
11 16 20 30725 335 51763 57212 6818 7108 8 filesystems net swaps
12 17 21 31521 3909 54 57990 68338 71459 9 loadavg partitions sys

Metricbeat also uses the normal proc filesystem so this needs to be mounted:

/root: mount -t procfs proc /proc

Confirm Information is Available:

/root: ls /proc
0 11 14 17 2 284 30725 324 3909 5 56221 57212 6 68338 7 72363 81714
1 12 15 18 20 28481 31521 326 4 51763 56937 57990 65594 69697 7108 73333 9
10 13 16 19 21 3 322 335 42931 54 56973 58529 6818 69866 71459 8 curproc

Once you have confirmed that the two proc filesystems are accessible then update /etc/fstab to mount the filesystems on startup.

proc /proc procfs rw 0 0
linproc /compat/linux/proc linprocfs rw 0 0

To load the kernel modules on startup the following needs to be added to /boot/loader.conf.local

verbose_loading="YES"
 linux_load="YES"
 linprocfs_load="YES"
 linux_enable="YES"

Reboot your system and check the kernel modules are loaded (as above) and the /proc and /compat/linux/proc directories are populated.

Configure Metricbeat

The Metricbeat  executable is located here:

/usr/local/sbin/metricbeat

The Metricbeat  config is located here:

/usr/local/etc/metricbeat.yml

We will amend the sample config so, if needed, copy the sample config file to the normal config file:

cp /usr/local/etc/metricbeat.yml.sample /usr/local/etc/metricbeat.yml

Edit the metricbeat.yml file as follows:

Comment out the Elasticsearch output settings:

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
 # Array of hosts to connect to.
 #hosts: ["localhost:9200"]

 

Uncomment the Logstash Output section and set the hostname to your Elasticsearch server:

output.logstash:
 # The Logstash hosts
 hosts: ["ubuntuelk.extelligence.it:5044"]

 

Configure logging to file (to aid in debug). The following is the logging section:

logging.level: info
logging.metrics.enabled: true
logging.to_syslog: false
logging.to_files: true
logging.files:
 path: /var/log/metricbeat
 name: metricbeat.log
 keepfiles: 7

Save the metricbeat.yml file and  create the log directory:

mkdir /var/log/metricbeat

Test the config file and correct any errors:

/usr/local/sbin/metricbeat -c /usr/local/etc/metricbeat.yml -e -configtest

 

Horray! Metricbeat is now ready to log to our Elasticstack instance.

 

Elasticsearch Index Template

As part of the Elasticbeats package a template for the Metricbeat index is provided. This should be installed on your Elasticstack server following the instructions on the Elasticstack website:

https://www.elastic.co/guide/en/beats/metricbeat/master/metricbeat-template.html

I installed the Elasticbeats package on the Elasticstack server as, by default, Elasticseach is not configured to allow connections from anything other than localhost.

Metricbeat Sample Dashboards:

Also as part of the Elasticbeats package a set of sample dashboards are provided. Install following the instructions on the Elasticstack website:

https://www.elastic.co/guide/en/beats/metricbeat/master/metricbeat-sample-dashboards.html

Again, having the Elasticbeats package installed on the Elasticstack server makes this easier.

Configuring Logstash on the Elasticstack server

This article presumes you have already installed the Elasticstack based upon Elasticstack (ELK), Suricata and pfSense Firewall – Part 2: Elasticstack Installation and Config

The Index and dashboards expect to find the Metricbeat data in a set of Metricbeat indexes. If you have followed the linked installation article Logstash will place the Metricbeat data into the default logstash indexes. To ensure that metricbeat data coming into logstash goes into separate indexes we can check the [type] field and see if the value is ‘metricsets’

 

Edit the /etc/logstash/conf.d/30-outputs.conf file to be the following:

output {
   if [type] == "metricsets" {
      elasticsearch {
         hosts => localhost
         index => "metricbeat-%{+YYYY.MM.dd}"
      }
   } else {
      elasticsearch {
         hosts => localhost
         index => "logstash-%{+YYYY.MM.dd}"
      }
   }
}

 

Configure Metricbeat to Autostart on pfSense boot

The method and information about the ‘why’ of this can be read in Elasticstack (ELK), Suricata and pfSense Firewall – Part 1: Elasticbeats and pfSense configuration as it was used to configure Filebeat to start automatically on pfSense boot.

On your pfSense instance add the following to /etc/rc.conf.local

metricbeat_enable=yes
metricbeat_conf=/usr/local/etc/metricbeat.yml

Reboot pfSense and check the processes are running with ps aux

 

Conclusion

 

With Metricbeat running on pfSense you should now have data appearing in the metricbeats indexes. Check out the sample dashboards and this article for a guide to creating some more.

Leave a Reply

Your email address will not be published. Required fields are marked *