Dynamic Log Orchestration via Rsyslog and syslog-ng Integration with Apache Kafka

The movement of telemetry data from edge devices to centralized stream processing engines represents a critical pillar of modern observability. As infrastructure scales, the traditional method of manually configuring static log paths becomes a bottleneck that introduces significant operational overhead and increases the probability of configuration drift. To address these challenges, architects leverage the dynamic routing capabilities of syslog daemons to stream events directly into Apache Kafka, a distributed event streaming platform. This architectural pattern enables real-time log ingestion, decoupling the log producer from the downstream consumers, and allowing for highly scalable, microservices-oriented observability pipelines.

Achieving this level of automation requires a deep understanding of how syslog daemons—specifically rsyslog and syslog-ng—interact with network protocols and the Kafka protocol. By implementing dynamic lookup mechanisms, administrators can move away from rigid, hardcoded rules and instead use the metadata of an incoming packet, such as the source IP address, to determine its destination topic within a Kafka cluster. This ensures that logs from various environments (development, staging, production) or different service tiers can be routed to specialized topics on the fly, providing the flexibility required for high-velocity, high-volume data environments.

Architecture of Rsyslog for Dynamic Kafka Routing

Rsyslog serves as the default syslog daemon on a vast majority of Linux distributions. It is characterized by its extreme speed and high degree of flexibility, making it an ideal candidate for acting as a high-performance log aggregator at the edge or in the middle of a pipeline. Unlike basic logging setups that merely write to local files, rsyslog can be configured to perform complex lookups to determine the destination of a message based on its properties.

In a sophisticated deployment, rsyslog is configured to listen for incoming syslog messages across multiple transport protocols. The most common configurations involve listening on both UDP (User Datagram Protocol) and TCP (Transmission Control Protocol) on the standard syslog port, which is typically port 514.

Transport Protocols and Port Configuration

The choice between UDP and TCP has a direct impact on the reliability and performance of the log stream:

• UDP (User Datagram Protocol): This is a connectionless protocol. While it offers lower latency and less overhead, it does not guarantee delivery. If a network partition occurs, packets may be lost without the sender's knowledge.
• TCP (Transmission Control Protocol): This is a connection-oriented protocol that provides reliable, ordered, and error-checked delivery of a stream of octets. This is generally preferred for mission-critical logging where data loss is unacceptable.

A functional configuration for an rsyslog receiver typically includes listening on port 514 for both protocols to ensure compatibility with a wide range of legacy and modern devices.

Dynamic Topic Selection via JSON Lookups

The most advanced feature of this rsyslog-to-Kafka pipeline is the use of a JSON-based lookup table. Instead of writing a new configuration line for every single IP address in a network, an administrator can maintain a JSON file that maps source IP addresses to specific Kafka topics.

When a syslog message arrives, rsyslog performs a lookup against this table. If the source IP is found, the message is tagged with the corresponding topic name. This mechanism provides several critical advantages for large-scale operations:

• Scalability: New clients can be added to the logging infrastructure simply by updating a JSON file, without needing to restart the rsyslog service or modify the core configuration logic.
• Reduced Complexity: It eliminates the need for hundreds of lines of if-then or action statements in the syslog configuration.
• Granular Control: Different teams or applications can have their own dedicated topics, ensuring that sensitive logs are isolated from general telemetry.

Implementation and Verification of the Data Pipeline

Once the rsyslog daemon is configured to perform these lookups, the next phase involves verifying that the data is actually reaching the Kafka broker and being placed into the correct topics. This requires a hands-on approach using command-line utilities to simulate real-world traffic.

Simulating Syslog Traffic

To validate the pipeline, one must simulate the behavior of a network device sending logs. This can be done using common Linux utilities like echo and nc (netcat), or by using the built-in device files in the /dev/ directory.

For UDP-based testing on port 514:
echo "<13>Sep 15 16:25:00 myhost myapp: This is a test from nc" | nc -u 127.0.0.1 514

For TCP-based testing on port 514:
echo "<34>1 2003-10-11T22:14:15.003Z localhost.elodina.net su - ID23 - a simple message" > /dev/tcp/192.168.66.66/5140

Note that the use of the > operator with /dev/tcp or /dev/udp allows the shell to open a socket and write the string directly to the destination, which is a highly efficient way to perform manual testing.

Consuming and Verifying Kafka Topics

To confirm that the logs are being routed correctly, a console consumer must be active on the Kafka side. This allows the operator to see the live stream of data as it arrives.

To consume from the beginning of a specific topic:
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic syslog --from-beginning

If the dynamic routing is working, the consumer will display the log message alongside its metadata. For example, a message sent from a specific host will appear in the topic designated for that host's IP address, proving the lookup logic is functional.

Troubleshooting and Verification Parameters

When verifying the setup, the following table outlines the key components and their roles in the verification process:

Advanced syslog-ng Integration and Consumption

While rsyslog is an excellent forwarder, syslog-ng is a powerful alternative that excels in complex log processing and, crucially, in consuming logs from Kafka back into the syslog ecosystem. This creates a bidirectional pipeline where logs can be sent from Kafka to other destinations like files or databases.

The Program-Based Workaround for Kafka Consumption

A notable technical detail in the syslog-ng ecosystem is that there is no "official" native Kafka source built into the core application. However, a highly efficient workaround exists using the program() source type. This method involves invoking a Kafka consumer script (such as the kafka-console-consumer.sh provided with the Kafka distribution) directly from within syslog-ng.

While this might seem unconventional, it is remarkably performant. In high-performance tests, this method has been shown to ingest between 50,000 and 100,000 messages per second on a single machine, even within resource-constrained virtual environments. This is achieved by leveraging Java-based consumers that can handle the high-throughput requirements of modern data streams.

Configuring syslog-ng for Kafka Ingestion

To implement this, a new configuration file should be created in the /etc/syslog-ng/conf.d/ directory. This keeps the main configuration clean and modular.

The following configuration snippet demonstrates how to use a program() source to pull logs from a Kafka topic and write them to a local file:

```
source skafka {
program("/root/kafka2.13-2.6.0/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic mytest");
};

destination d_fromkafka {
file("/var/log/fromkafka");
};

log {
source(skafka);
destination(dfromkafka);
};
```

This configuration establishes a flow where syslog-ng acts as a client to Kafka. It executes the consumer command, captures the standard output, and redirects it to a local file for permanent storage or further analysis.

Performance Tuning and Batching Optimization

When dealing with high-volume telemetry, the efficiency of the data transfer is determined by how the syslog daemon handles "flushing" or sending data to the destination. This is particularly critical when using the kafka() or kafka-c() destinations in syslog-ng.

Understanding Batch-Lines and Latency

The batch-lines() option is a critical parameter for optimizing throughput. It determines how many individual log lines are collected in memory before being sent to Kafka as a single batch.

• High batch-lines() Value: Increasing this number improves overall throughput because there is less overhead associated with individual network requests. However, this comes at the cost of increased latency, as logs will sit in a buffer waiting for the batch to fill.
• Low batch-lines() Value: Setting this to 1 (the default) ensures that messages are sent almost immediately, providing the lowest possible latency, but it can significantly decrease total throughput due to the high volume of small network packets.

Batch Flushing Logic and Lifecycle

The interaction between the queue state and the batching mechanism is complex and must be understood to prevent data loss or extreme latency. syslog-ng OSE (Open Source Edition) follows specific rules for when to flush a batch:

1. Active Processing: If the system is actively receiving a steady stream of logs and the batch-lines() limit is reached, the batch is flushed immediately. In this state, the batch-timeout() setting is ignored in favor of the line limit.
2. Empty Queue: If the log stream pauses, syslog-ng will only flush the remaining messages in the buffer once the batch-timeout() period has expired. If batch-timeout() is disabled, the queue is flushed immediately upon becoming empty.
3. Shutdown and Reload: To ensure data integrity, syslog-ng is designed to automatically flush all unsent messages to the destination when the service is stopped or reloaded, or when a network connection to a client is closed.

Comparison of syslog-ng Kafka Destination Options

The following table provides a detailed look at the operational parameters for Kafka destinations:

Architectural Conclusion and Strategic Analysis

The integration of rsyslog/syslog-ng with Apache Kafka represents a transition from "static logging" to "event streaming." By utilizing the dynamic lookup capabilities of rsyslog, organizations can build an infrastructure that is both highly flexible and extremely scalable. The ability to route logs based on source IP addresses via JSON mapping allows for a "plug-and-play" approach to device onboarding, which is essential in large-scale cloud and edge computing environments.

Furthermore, the ability to use syslog-ng as a consumer through the program() source mechanism provides a powerful method for "closing the loop" in an observability pipeline. This allows organizations to ingest logs from Kafka back into traditional syslog workflows, enabling a hybrid approach where modern, stream-based architectures can still interoperate with legacy monitoring and alerting tools.

Ultimately, the success of such a pipeline depends on the careful calibration of batching parameters. An architect must weigh the business requirement for real-time visibility (low latency) against the technical requirement for high-volume data ingestion (high throughput). By mastering the interaction between batch-lines(), batch-timeout(), and the underlying transport protocols, engineers can build robust, resilient, and high-performance telemetry pipelines capable of handling the massive data demands of modern digital ecosystems.

Sources

1. Seynur Blog - Streaming rsyslog events to Kafka
2. GitHub - syslog-kafka repository by elodina
3. syslog-ng Blog - Consuming logs from a Kafka topic
4. syslog-ng Admin Guide - Kafka Destination Options