Elasticstack (ELK) and pfSense Firewall – IP Traffic Statistics with Netflow

Elasticstack (ELK) and pfSense Firewall – IP Traffic Statistics with Netflow

Introduction

In Logstash V5.6 a Netflow module was introduced to provide the  collection, normalisation, and visualisation of network flow data. This article, which details the configuration of Elasticstack as a Netflow collector and pfSense as a Netflow exporter, is a follow-on from the previously published articles. Only the previous Part 2 article, which deals with configuring Elasticstack on an Ubuntu server, is required in order to implement the Netflow IP Traffic Statistics covered in this article.

 

Netflow

Netflow was introduced by Cisco in their routers to provide the capability to provide detailed statistics on IP traffic flows. The Netflow process on the router tracks IP flows and periodically exports Netflow datagrams containing details about each flow to a Netflow collector which is able to store and process the information to provide statistical and visual representations of the traffic flowing in to and out of router interfaces.

Configuring the Elasticstack Logstash Netflow Module

This presumes you have set up Elasticstack as per Part 2 of the original series of articles the Logstash Netflow Module page provides detailed instructions on configuring the module but the basic steps are as follows:

Installing the module’s index patterns, visualisations and dashboards:

Login to the Ubutu Server that is hosting your elasticStack

Run the following command in the Logstash installation directory:

sudo bin/logstash –modules netflow –setup

This is a one-time action which installs the relevant indexes, visualisations and dashboards into your Elasticstack instance. The above command will not exit once complete so check in Kibana, using a browser, that the relevant objects have been created in the management. If they exist (as shown below) and the above command has not displayed any errors CTRL-C the above command to exit.

Configuring Logstash to load the module

Edit the logstash.yml config file

sudo joe /etc/logstash/logstash.yml

Insert the following. There is a modules section detailed in the file so to keep it neat perhaps insert it there

modules:
  - name: netflow
    var.input.udp.port: 2055

Restart logstash and tail the logstash log to ensure there are no errors:

systemctl restart logstash.service
tail -f /var/log/logstash/logstash-plain.log

You should see something to the similar in the log:

[logstash.pipeline ] Pipeline main started
[logstash.inputs.udp ] Starting UDP listener {:address=>"0.0.0.0:2055"}
[logstash.inputs.udp ] UDP listener started {:address=>"0.0.0.0:2055", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}

Your Logstash process is now listening patiently waiting for Netflow data!

Configuring pfSense to export Netflow data

pfSense requires a the softflowd package to be loaded in order to add the functionality to export Netflow data. Install the softflowd package from your pfSense webgui under the system…packages menu.

Under the Services menu enter the softflowD configuration, pick the Interface(s) you want to be monitored and enter the host and port information for your Elasticstack server running logstash (Note the Host MUST be an IP address 🙁 )

Your pfSense firewall should now start exporting Netflow data about active flows.

Verify data is being indexed

Use the Kibana discover tool and ensure the index is selected to be netflow-*

If correctly configured you will have records in the index similar to the below:

Dashboards and Pretty Pictures

Within the Dashboard section within Kibana you should now be able to access a wealth of information about the network flows that are entering and exiting your pfSense firewall interfaces.

11 Replies to “Elasticstack (ELK) and pfSense Firewall – IP Traffic Statistics with Netflow”

  1. root@OMV-MediaServer:/usr/share/logstash# sudo bin/logstash –modules netflow –setup
    WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using –path.settings. Continuing using the defaults
    ERROR: Unknown command ‘–modules’

    See: ‘bin/logstash –help’

    Any ideas?

    1. I think it looks like (from the next comment) you resolved this. Was is an issue in the tutorial around the installation of the package?

  2. Meh

    root@OMV-MediaServer:/usr/share/logstash# bin/logstash –modules netflow –setup -M netflow.var.input.udp.port=NNNN
    WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using –path.settings. Continuing using the defaults
    Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
    [FATAL] 2018-01-04 15:22:37.677 [LogStash::Runner] runner – An unexpected error occurred! {:error=>#, :backtrace=>[“/usr/share/logstash/logstash-core/lib/logstash/settings.rb:299:in `coerce'”, “/usr/share/logstash/logstash-core/lib/logstash/settings.rb:256:in `set'”, “/usr/share/logstash/logstash-core/lib/logstash/modules/logstash_config.rb:45:in `get_setting'”, “/usr/share/logstash/logstash-core/lib/logstash/modules/logstash_config.rb:59:in `setting'”, “(erb):4:in `result'”, “org/jruby/RubyKernel.java:1079:in `eval'”, “/usr/share/logstash/vendor/jruby/lib/ruby/1.9/erb.rb:838:in `result'”, “/usr/share/logstash/logstash-core/lib/logstash/modules/logstash_config.rb:91:in `config_string'”, “/usr/share/logstash/logstash-core/lib/logstash/modules/scaffold.rb:49:in `config_string'”, “/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:88:in `pipeline_configs'”, “org/jruby/RubyArray.java:1613:in `each'”, “/usr/share/logstash/logstash-core/lib/logstash/config/modules_common.rb:56:in `pipeline_configs'”, “/usr/share/logstash/logstash-core/lib/logstash/runner.rb:276:in `execute'”, “/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'”, “/usr/share/logstash/logstash-core/lib/logstash/runner.rb:204:in `run'”, “/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'”, “/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'”]}
    root@OMV-MediaServer:/usr/share/logstash#

    Can not get the module to install.

  3. No results found
    Unfortunately I could not find any results matching your search. I tried really hard. I looked all over the place and frankly, I just couldn’t find anything good. Help me, help you. Here are some ideas:

    Expand your time range
    I see you are looking at an index with a date field. It is possible your query does not match anything in the current time range, or that there is no data at all in the currently selected time range. Click the button below to open the time picker. For future reference you can open the time picker by clicking on the time picker button in the top right corner of your screen.

    I give up. Thanks anyway.

    1. Check the Logstash logs on the server to ensure that data is being received. Also try the following command in the Kibana dev Tools area to check you have indexes, their names and sizes (or none)

      GET /_cat/indices

    1. The above is one of the default dashboards from the Elastic Netflow module which should have been installed as part of the above tutorial. It *may* be that GeoIP data is missing – use Discover to check that you’ve got data in the geoip.ip field and then also that there is data similar to this in the geoip.location

      {
      “lon”: -122.0574,
      “lat”: 37.419200000000004
      }

      If there is then there should be no reason why you would not see a similar visualisation to the above, providing your traffic is coming from a variety of sources – The geoIP.ip is taken the source ip of the flow.

Leave a Reply

Your email address will not be published. Required fields are marked *