Deployment Architecture for the Rsyslog Operations Stack Initiative (ROSI) and Grafana Observability

The integration of rsyslog into a modern observability ecosystem represents a significant shift from traditional, isolated log storage toward a unified, production-ready monitoring strategy. As of April 2026, the emergence of the Rsyslog Operations Stack Initiative, commonly referred to as ROSI, provides a standardized blueprint for engineers who previously struggled with the fragmentation of the "LGTM" (Loki, Grafana, Tempo, Mimir) stack when attempting to incorporate legacy or high-throughput syslog streams. This architectural evolution focuses on transforming rsyslog from a mere message-passing engine into the foundational pillar of a centralized log collection and monitoring framework. The introduction of the ROSI Collector into the rsyslog main branch via pull request #6325 marks a transition from experimental configuration to a documented, reference-grade deployment. For the systems administrator or DevOps engineer, this means moving away from the manual, often error-prone process of stitching together individual binaries and toward a coherent, containerized environment that is designed for immediate operational visibility.

The ROSI Collector Architecture and Component Integration

The ROSI Collector is not merely a demonstration of capability but a production-oriented observability stack engineered for centralized log management. It is delivered through a Docker Compose deployment, providing a structured approach to managing the lifecycle of various telemetry-gathering services. By utilizing Docker Compose, the stack ensures that the networking and volume persistence required for high-volume logging are pre-configured, reducing the "configuration drift" that often plagues manual deployments of observability tools.

The architecture of the ROSI Collector integrates several high-performance components into a singular, cohesive unit:

  • rsyslog as a centralized log receiver: Acts as the primary ingestion point for all system and application logs, capable of handling diverse protocols and formats.
  • Grafana Loki: Serves as the horizontally scalable, multi-tenant log aggregation system, optimized for high-volume log storage and efficient querying.
  • Prometheus: Functions as the time-series database for metrics collection, monitoring the health of the logging pipeline itself.
  • Grafana: Provides the visualization layer, featuring preconfigured dashboards that render the ingested data into actionable insights.
  • Traefik: Operates as the edge reverse proxy, managing incoming traffic and providing automatic TLS termination via Let’s Encrypt to ensure all log transmissions remain encrypted.

The integration of Traefik is particularly critical for the security posture of the stack. By automating certificate management, the architecture enforces industry-standard encryption for incoming syslog streams without requiring manual intervention for certificate renewals. This setup ensures that the transition from raw log generation to visual representation in Grafena is seamless and secure.

Operational Properties and Secure Transport Protocols

A primary differentiator between a simple logging demo and the ROSI Collector is its adherence to operational best practices. The stack is built as a reference deployment, meaning its configurations are modeled after real-world production environments rather than minimal, stripped-down examples. This includes the implementation of advanced transport layers and security protocols.

The capability for secure syslog reception is a cornerstone of the ROSI deployment. The stack supports the implementation of TLS (RFC 5425), which is essential for protecting sensitive log data as it traverses untrusted networks. Furthermore, the support for mutual TLS (mTLS) allows for a high-assurance environment where both the client and the server must present valid certificates, preventing unauthorized entities from injecting fraudulent log entries into the centralized repository. To facilitate this, the ROSlyog ROSI deployment includes dedicated certificate generation helpers, which remove the significant operational burden of managing a private PKI (Public Key Infrastructure) for the initial setup.

Beyond security, the stack is designed for immediate observability. The inclusion of predefined Grafana dashboards means that upon the first successful deployment, operators are immediately presented with visibility into:

  • Log flow metrics: Tracking the volume of logs arriving at the rsyslog receiver.
  • System state: Monitoring the health and resource utilization of the underlying Docker containers.
  • Operational metrics: Observing the performance of the forwarding pipelines and storage backends.

To further assist in the maintenance of this ecosystem, the ROSI deployment includes a suite of operational helpers. These are specialized scripts designed for:

  • Certificate handling: Managing the lifecycle of the TLS certificates used for secure transport.

  • Stack status checks: Verifying the availability of all components within the Docker Compose network.

  • Prometheus target management: Ensuring that the metrics collection engine is correctly identifying and scraping all relevant endpoints.

Advanced Configuration of Rsyslog Modules and Rulesets

To achieve a robust observability pipeline, the rsyslog configuration must be meticulously tuned to handle both local and remote log streams. A standard Ubuntu installation includes rsyslog by default, and its configuration is modularized through files located in /etc/rsyslog.d/. This modularity allows engineers to define specific rulesets for remote log reception without disrupting the default local system logging logic.

Effective configuration requires the loading of specific modules to enable various input types. A production-grade rsyslog.conf must include modules for local socket support, UDP, and TCP.

The following module definitions are foundational for a complete logging setup:

  • imuxsock: Provides the necessary support for local system logging via the /dev/log socket.

  • imudp: Enables the reception of syslog messages over the UDP protocol, which is useful for low-overhead, high-speed log ingestion.

  • imtcp: Enables the reception of syslog messages over the TCP protocol, providing the reliability required for critical log streams.
  • imklog: Facilitates kernel logging support, including the ability to enable non-kernel klog messages.

A critical aspect of high-volume log forwarding is the configuration of action queues. When forwarding logs from a local rsyslog instance to a centralized collector, the use of a linked-list queue type is recommended to prevent data loss during periods of high network latency or downstream service unavailability.

The following configuration fragment demonstrates a robust forwarding rule:

*.* action(type="omfwd" protocol="tcp" target="127.0.0.1" port="1514" Template="RSYSLOG_SyslogProtocol23Format" TCP_Framing="octet-counted" KeepAlive="on" action.resumeRetryCount="-1" queue.type="linkedlist" queue.size="50000")

In this configuration, action.resumeRetryCount="-1" ensures that the forwarder will retry indefinitely, while queue.size="50000" provides a substantial buffer to absorb spikes in log volume. The use of RSYSLOG_SyslogProtocol23Format ensures compatibility with modern logging standards, and TCP_Framing="octet-counted" is essential for maintaining the integrity of the message stream over TCP.

For more complex environments, rsyslog allows for sophisticated rulesets to filter between local and external messages. For instance, an administrator might choose to forward only high-priority messages from a specific host to a central Grafana instance, while allowing all other logs to remain local.

:hostname, contains, "grafanapi" *.notice @@(o)127.0.0.1:6514;RSYSLOG_SyslogProtocol23Format :hostname, !contains, "grafanapi" *.* @@(o)127.0.0.1:6514;RSYSLOG_SyslogProtocol23Format

In this example, logs from the host containing "grafanapi" are only forwarded if their severity is at least at the notice level, whereas all other hosts have their entire log stream forwarded regardless of severity.

Integrating Grafana Alloy for Advanced Labeling and Querying

While rsyslog handles the ingestion and forwarding of logs, the observability pipeline's utility in Grafana depends heavily on how those logs are labeled. A common challenge in modern observability is the "labeling problem," where logs are ingested but lack the metadata (such as host, service, PID, or severity) required to perform efficient, granular queries.

Grafana Alloy (the successor to Promtail) serves as a critical intermediary in this pipeline. The configuration of Alloy can be notoriously difficult, particularly when attempting to map syslog-specific fields into Loki-compatible labels. A well-structured configuration must utilize loki.relabel components to transform the incoming syslog attributes into searchable labels.

A functional Alloy configuration for a syslog-to-Loki pipeline requires a loki.write component to define the destination and a loki.source.syslog component to listen for incoming streams. The following configuration demonstrates how to implement a labelmap action to extract metadata from the syslog headers:

```
logging {
level = "warn"
}

// Configure a loki.write component to send logs to a local Loki instance.
loki.write "default" {
endpoint {
url = "http://localhost:3100/loki/api/v1/push"
}
}

// Configure relabeling to transform syslog metadata into useful Loki labels.
loki.relabel "syslog" {
forwardto = []
rule {
action = "labelmap"
regex = "
syslog(.+)"
}
rule {
action = "labelmap"
regex = "message_(.+)"
}
}

// Configure a loki.source.syslog component to receive syslog messages.
loki.source.syslog "default" {
listener {
address = "0.0.0.0:1514"
// Add a 'job' label to all logs received by this component.
labels = {"job" = "syslog"}
}
relabelrules = loki.relabel.syslog.rules
// Forward received logs to the loki.write component.
forward
to = [loki.write.default.receiver]
}
```

In this configuration, the loki.relabel component uses regular expressions to identify patterns in the syslog stream (such as __syslog_ or message_) and maps them directly to Loki labels. This transformation is what enables an engineer to execute a query in Grafana like {job="syslog", service="nginx"} rather than performing expensive, full-text searches across the entire log index.

Deployment Scopes and Future Evolution

The current scope of the ROSI Collector is intentionally focused. It targets environments that are predominantly VM-based or utilize single-host Docker deployments. These environments are prioritized because they represent a massive portion of the enterprise landscape and favor solutions that are understandable, inspectable, and maintain low operational complexity.

The development roadmap for ROSI suggests a clear path toward container orchestration. While Kubernetes support is a highly anticipated extension, it was intentionally excluded from the initial merge into the rsyslog main branch. The architectural philosophy here is to establish a solid, well-understood baseline for traditional deployments before tackling the much higher complexity of Kubernetes-native log collection.

The evolution of this stack will likely involve:

  • Kubernetes-native deployment models.
  • Enhanced integration with the wider Grafana ecosystem.
  • Further expansion of the documentation to bridge the gap between the evolving codebase and the deployment guides.

Analysis of Observability Pipeline Integrity

The convergence of rsyslog and the Grafana observability stack represents a significant maturation of the logging industry. Historically, the industry has been split between "legacy" syslog-based systems, which are highly reliable but difficult to query, and "modern" cloud-native observability tools, which are highly queryable but often struggle with the sheer volume and variety of traditional syslog formats.

The ROSI Collector bridges this gap by providing a standardized, production-ready configuration that treats rsyslog as a first-class citizen within the modern observability pipeline. By integrating Traefik for security, Prometheus for monitoring, and Alloy for intelligent labeling, the architecture moves beyond simple log aggregation into true observability. The ability to perform complex, multi-dimensional queries in Grafana—enabled by the meticulous relabeling of syslog attributes—transforms logs from a forensic tool used only after a failure into a proactive monitoring asset. As this stack evolves toward Kubernetes support, it will likely become the standard reference architecture for any organization looking to unify their legacy infrastructure with their modern, containerized workloads.

Sources

  1. Rsyslog Official News
  2. Alexandre Deverteuil - Syslog Relay for Loki
  3. NWMichl Blog - Configuring Grafana Alloy
  4. Grafana Community - Rsyslog/Promtail/Loki Help
  5. Grafana Dashboards - Syslog Dashboard

Related Posts