Integrating pfSense with the ELK Stack for Advanced Network Security Monitoring and OSCP Lab Environments

The modern landscape of network security demands more than just a perimeter firewall; it requires a comprehensive visibility strategy where telemetry is centralized, parsed, and visualized in real-time. For security professionals, researchers, and those pursuing the Offensive Security Certified Professional (OSCP) certification, the combination of pfSense, Snort, and the ELK Stack (Elasticsearch, Logstash, Kibana) represents a gold standard for building a transparent and defensible network environment. pfSense, an open-source firewall and router distribution based on FreeBSD, provides the foundational routing and filtering capabilities, while Snort adds an Intrusion Detection and Prevention System (IDS/IPS) layer. However, the raw logs generated by these systems are often voluminous and difficult to analyze in their native state. This is where the ELK Stack becomes critical, transforming unstructured syslog and Netflow data into actionable intelligence through a sophisticated pipeline of ingestion, normalization, and visualization.

Deploying the pfSense Foundation

The first phase of establishing this security architecture begins with the deployment of pfSense. As a FreeBSD-based distribution, pfSense is engineered for high performance and stability, making it suitable for both small-scale home labs and enterprise-grade deployments.

The installation process begins with the acquisition of the official pfSense installer. For the vast majority of modern hardware, the AMD64 architecture is the required selection. Once the image is obtained, it must be written to a bootable medium, such as a USB drive, utilizing imaging software like Rufus or Etcher to ensure the boot sector is correctly configured.

Upon booting the hardware from the USB medium, the installer facilitates the configuration of the network interfaces. A critical step in this process is the definition of the WAN (Wide Area Network) and LAN (Local Area Network) interfaces.

• The WAN interface is designated to handle external connectivity, interfacing with the internet service provider or an upstream router.
• The LAN interface serves as the gateway for the internal network, ensuring that all traffic from local devices is routed through the firewall for inspection.

In environments with multiple network interface cards (NICs), pf uma automatically detects available hardware, allowing the administrator to assign specific ports to the WAN and LAN roles during the initial setup phase. This separation is fundamental to the security posture, as it allows the firewall to act as a strict gatekeeper between untrusted external traffic and the trusted internal environment.

Architectural Integration with the OSCP Lab

For those utilizing this setup within an OSCP (Offensive Security Certified Professional) lab, the integration of pfSense serves a dual purpose: it provides a secure environment for practicing penetration testing techniques and simulates a real-world corporate network where security monitoring is active.

By integrating Snort and the ELK Stack, the lab evolves from a simple set of isolated VMs into a high-fidelity monitoring environment. Snort analyzes network traffic against a database of known signatures to detect malicious activity, while the ELK Stack provides the historical record and visualization of these alerts. This setup allows a student or professional to see exactly how their attacks appear to a defender, providing a closed-loop learning experience where the "attacker" can analyze the "defender's" logs in real-time.

The ELK Stack Ingestion Pipeline: Logstash Configuration

The ELK Stack relies on Logstash as the primary data processing engine. Logstash is responsible for receiving data from pfSense, filtering it, and sending it to Elasticsearch for indexing.

Configuring Syslog Inputs

To ingest syslog data from pfSense, Logstash must be configured to listen on specific ports. A standard implementation involves creating an input configuration file located at /etc/logstash/conf.d/01-inputs.conf. This file defines how Logstash accepts data over the network.

The following commands are used to retrieve and verify the input configuration:

bash elk@stack:~$ cd /etc/logstash/conf.d/ elk@stack:/etc/logstash/conf.d$ sudo wget https://raw.githubusercontent.com/psychogun/ELK-Stack-on-Ubuntu-for-pfSense/master/etc/logstash/conf.d/01-inputs.conf elk@stack:/etc/logstash/conf.d$ more 01-inputs.conf

The content of the 01-inputs.conf file ensures that the system is listening for syslog data on both TCP and UDP protocols at port 5140:

text input { tcp { type => "syslog" port => 5140 } } input { udp { type => "syslog" port => 5140 } }

This dual-protocol approach is essential because different network devices or pfSense packages may utilize different transport layers for their logging mechanisms.

Implementing Syslog Filters and Tagging

Once the data is received, it is unstructured. The filter stage, managed by /etc/logstash/conf.d/10-syslog.conf, is used to categorize the logs based on the source IP address. This allows the administrator to differentiate between multiple pfSense instances in a complex environment.

The configuration involves the following steps:

bash elk@stack:/etc/logstash/conf.d$ sudo wget https://raw.githubusercontent.com/psychogun/ELK-Stack-on-Ubuntu-for-pfSense/master/etc/logstash/conf.d/10-syslog.conf elk@stack:/etc/logstash/conf.d$ more 10-syslog.conf

The filtering logic applies specific tags to the logs based on the host IP:

• If the host matches 192.168.40.1, the tag pfsense and Ready are added.
• If the host matches 172.2.22.1, the tag pfsense-2 and Ready are added.
• If no match is found and the Ready tag is absent, a generic syslog tag is applied.

The final stage of the filter removes the Ready tag to clean up the metadata and applies a Grok pattern to parse the raw message into structured fields:

text filter { if "syslog" in [tags] { grok { match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" } add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", ... ] } } }

Grok is a critical component of the ELK Stack as it converts a string of text into a queryable database entry, allowing users to search for specific timestamps, hostnames, or process IDs.

Advanced Log Parsing with Grok

For deeper analysis of pfSense firewall logs, a more complex Grok pattern is required to handle the specific format of firewall events, including rule IDs, tracker numbers, and IP addresses. This is often defined in configuration files such as 11-pfsense.conf.

The following pattern represents the highly granular parsing required to extract firewall data:

text PFSENSE_LOG_DATA (%{INT:rule}),,,(%{INT:tracker}),(%{WORD:iface}),(%{WORD:reason}),(%{WORD:action}),(%{WORD:direction}),(%{INT:ip_ver}), PFSENSE_IP_SPECIFIC_DATA (%{PFSENSE_IPv4_SPECIFIC_DATA}|%{PFSENSE_IPv6_SPECIFIC_DATA}) PFSENSE_IPv4_SPECIFIC_DATA (%{BASE16NUM:tos}),,(%{INT:ttl}),(%{INT:id}),(%{INT:offset}),(%{WORD:flags}),(%{INT:proto_id}),(%{WORD:proto}), PFSENSE_IPv4_SPECIFIC_DATA_ECN (%{BASE16NUM:tos}),(%{INT:ecn}),(%{INT:ttl}),(%{INT:id}),(%{INT:offset}),(%{WORD:flags}),(%{INT:proto_id}),(%{WORD:proto}), PFSENSE_IPv6_SPECIFIC_DATA (%{BASE16NUM:class}),(%{DATA:flow_label}),(%{INT:hop_limit}),(%{WORD:proto}),(%{INT:proto_id}), PFSENSE_IP_DATA (%{INT:length}),(%{IP:src_ip}),(%{IP:dest_ip}), PFSENSE_PROTOCOL_DATA (%{PFSENSE_TCP_DATA}|%{PFSENSE_UDP_DATA}|%{PFSENSE_ICMP_DATA}|%{PFSENSE_CARP_DATA}) PFSENSE_TCP_DATA (%{INT:src_port}),(%{INT:dest_port}),(%{INT:data_length}),(%{WORD:tcp_flags}),(%{INT:sequence_number}),(%{INT:ack_number}),(%{INT:tcp_window}),(%{DATA:urg_data}),(%{DATA:tcp_options}) PFSENSE_UDP_DATA (%{INT:src_port}),(%{INT:dest_port}),(%{INT:data_length}) PFSENSE_ICMP_DATA (%{PFSENSE_ICMP_TYPE}%{PFSENSE_ICMP_RESPONSE}) PFSENSE_ICMP_TYPE

This detailed pattern allows the ELK stack to capture everything from the TCP flags and sequence numbers to the specific ICMP response types, providing a forensic-level view of every packet that hits the firewall.

Implementing Netflow Monitoring with softflowd

While syslogs provide an event-based view (e.g., "this packet was blocked"), Netflow provides a session-based view (e.g., "this IP transferred 50MB of data over 10 minutes"). To achieve this in pfSense, the softflowd package is utilized.

Installation and Configuration of softflowd

The installation is performed through the pfSense web interface by navigating to System > Package Manager > Available Packages and installing softflowd. Once installed, the service is configured under Services > softflowd.

The following configuration parameters are required for successful integration with Logstash:

A critical security note is that the communication between softflowd and Logstash for Netflow is not encrypted, meaning the flow data travels in cleartext across the management network.

GeoIP Integration for Enhanced Visualization

To add geographical context to the firewall logs, the GeoLite2 database is integrated into the Logstash directory. This allows the ELK stack to map an IP address to a physical location, which is invaluable for identifying the origin of external attacks.

The installation process involves downloading and extracting the GeoLite2 database into the Logstash directory:

bash elk@stack:/etc/logstash$ sudo tar xvzf GeoLite2-City_20200428.tar.gz elk@stack:/etc/logstash$ sudo mv GeoLite2-City_20200428/GeoLite2-City.mmdb .

After the .mmdb file is moved to the root of the Logstash configuration directory, the temporary folder is removed to maintain system cleanliness:

bash elk@stack:/etc/logstash$ sudo rm -rf GeoLite2-City_20200428

Kibana Visualization and Dashboard Management

The final stage of the ELK implementation is the creation of visualizations in Kibana. This transforms raw indexed data into graphical dashboards that provide an immediate understanding of the network security posture.

Importing Visualization Objects

To establish a baseline of monitoring, users can import a set of 31 pre-defined objects from the project's visualization repository. These objects define the charts, maps, and tables used in the dashboards.

The primary dashboards to be imported from the GitHub repository include:

• Dashboard - Firewall - External Block.json: Focuses on traffic that is being actively dropped by the firewall rules.
• Dashboard - Firewall - External Pass.json: Focuses on traffic that is permitted into the network, useful for identifying unauthorized but permitted access.
• Dashboard - pfSense.json: An overall summary dashboard that aggregates various syslog events from the firewall.

These dashboards allow the administrator to mix and match visualizations to focus on specific metrics, such as the most frequent source IPs of blocked traffic or the most targeted ports on the WAN interface.

System Tuning and Performance Optimization

Running a full ELK stack can be resource-intensive, particularly for home users or those in a lab environment. Tuning the Java Virtual Machine (JVM) options is necessary to prevent the system from crashing due to memory exhaustion.

The primary files for tuning are:

• /etc/elasticsearch/jvm.options
• /etc/logstash/jvm.options

Adjusting the heap size in these files ensures that Elasticsearch and Logstash have sufficient memory to handle the ingestion of high-volume logs from Snort and pfSense without causing system instability.

Conclusion: Comprehensive Analysis of the Security Posture

The integration of pfSense, Snort, and the ELK Stack creates a potent security ecosystem. By moving from simple log files to a centralized analysis platform, an organization gains the ability to perform proactive threat hunting rather than reactive log auditing.

The synergy between these tools is evident in the data flow: pfSense handles the routing and basic filtering, Snort identifies complex attack signatures, softflowd provides the volumetric session data, and the ELK stack synthesizes this into a visual narrative. For the OSCP student, this environment is invaluable because it demonstrates the "blue team" side of security—how alerts are generated, how they are parsed via Grok, and how they are presented to a security analyst via Kibana.

The ultimate effectiveness of this system depends on the continuous review of dashboards. By analyzing the "External Block" and "External Pass" dashboards, an administrator can identify emerging attack patterns, tune Snort rules to reduce false positives, and refine pfSense firewall rules to harden the network. This holistic approach to monitoring transforms the network from a black box into a transparent environment where every single packet is accounted for and every security event is indexed for future forensic analysis.

Sources

1. pfSense, OSCP, Snort & ELK: A Step-by-Step Guide
2. ELK pfSense 2.3 Working
3. ELK stack on Ubuntu with pfSense