Orchestrating the Elastic Stack on Red Hat OpenShift via Elastic Cloud on Kubernetes (ECK)

The modern enterprise landscape is defined by the management of hundreds or even thousands of containers, a scale that has transitioned from a luxury to a standard operational requirement for most organizations. As infrastructures grow in complexity, the necessity for a centralized, robust, and scalable observability framework becomes critical. Red Hat OpenShift, as a premier Kubernetes container platform, provides the foundation for these workloads, but the sheer volume of logs and metrics generated by such an environment necessitates a sophisticated telemetry solution. The Elastic Stack (formerly known as ELK) emerges as the industry standard for this purpose, providing a comprehensive suite for search, analytics, and visualization. To bridge the gap between the raw power of the Elastic Stack and the orchestration capabilities of OpenShift, Elastic Cloud on Kubernetes (ECK) serves as the official Operator. This operator is designed to simplify the entire lifecycle of Elasticsearch and Kibana, transforming what would be a manual, error-prone installation process into a streamlined, automated deployment that ensures best practices are baked into the infrastructure from day one.

The integration of the Elastic Stack into OpenShift is not merely about installation but about establishing a holistic observability ecosystem. By utilizing Elastic Observability, administrators can store and analyze logs and metrics from the OpenShift environment alongside other infrastructure monitoring data. This creates a unified view of the entire stack, allowing for rapid root-cause analysis and proactive system health management. This architectural synergy ensures that as the OpenShift cluster scales, the monitoring capabilities scale in tandem, preventing the observability gaps that often lead to catastrophic system outages.

The Architecture of the ELK Stack

The Elastic Stack is composed of three primary pillars, each serving a distinct role in the data pipeline. Understanding these components is essential for configuring a performant deployment on OpenShift.

Elasticsearch: The Analytics Engine

Elasticsearch serves as the core engine of the entire stack. It is a distributed, RESTful search and analytics engine that provides real-time search capabilities across all data types, including structured, unstructured, and numerical data.

The technical foundation of Elasticsearch allows it to index data in a manner that maximizes the speed of search and retrieval. This is achieved through a highly scalable architecture where, as the volume of data expands, the system can deploy additional nodes to meet the increasing demand. Beyond simple storage, Elasticsearch aggregates data to identify trends and patterns, which is invaluable for analyzing security events or performing website search operations. Furthermore, it supports high-availability alerting mechanisms that can trigger notifications via email, webhooks, Jira, Microsoft Teams, and Slack, ensuring that system administrators are notified of anomalies in real-time.

Logstash: The Data Pipeline

Logstash acts as the aggregation and transformation layer. Its primary responsibility is to collect data from diverse sources, transform that data into a usable format, and then ship it to Elasticsearch for indexing.

The operational flow of Logstash involves receiving inputs, filtering them, and then producing outputs. Inputs are typically sourced from files, syslogs, or lightweight log shippers known as Beats. Once the data is received, Logstash applies filters to clean or restructure the information, ensuring that the data stored in Elasticsearch is consistent and queryable. While the primary output is Elasticsearch, Logstash is flexible enough to generate outputs to files, graphics, or various other formats.

Kibana: The Visualization Layer

Kibana provides the user interface for the Elastic Stack. It is the window through which administrators and analysts interact with the data stored in Elasticsearch.

Kibana allows users to explore data through intuitive dashboards and provides the necessary tools to manage and monitor the entire ELK cluster. It gives visual shape to the raw data, turning complex logs and metrics into actionable insights. By providing a centralized management console, Kibana enables the navigation of the entire stack, making it the primary tool for observing system health and performing deep-dive forensics.

Technical Prerequisites and Cluster Preparation

Before deploying the Elastic Stack on OpenShift, certain administrative and system-level requirements must be met to ensure stability and performance.

Administrative Privileges

The deployment process requires high-level permissions. The user performing the installation must be a system:admin user or possess specific privileges that allow for the creation of:

• Projects
• Custom Resource Definitions (CRDs)
• Role-Based Access Control (RBAC) resources at the cluster level

Without these permissions, the ECK Operator cannot manage the underlying Kubernetes resources required to orchestrate the stateful sets of Elasticsearch.

Virtual Memory Optimization

A critical technical requirement for Elasticsearch is the configuration of the host's virtual memory. By default, Elasticsearch utilizes a mmapfs directory to store its indices efficiently. However, the default operating system limits on mmap counts in most Linux distributions are typically too low for the demands of a production database.

If the kernel settings are not adjusted, the system is likely to encounter out-of-memory (OOM) exceptions, leading to pod crashes and data instability. For production workloads, it is strongly recommended to increase the kernel setting vm.max_map_count to 262144. This adjustment ensures that the operating system can handle the large number of memory-mapped files created by Elasticsearch.

Deploying the Elastic Stack via ECK

The use of the ECK Operator allows for the orchestration of a stateful deployment in minutes, incorporating built-in best practices.

Initial Setup of Elasticsearch and Kibana

The primary goal of using ECK on OpenShift is to automate the setup, upgrades, and scaling of the stack. By deploying the operator, administrators can define the desired state of their Elasticsearch and Kibana clusters through YAML manifests, which the operator then realizes on the OpenShift platform. This eliminates the need for manual configuration of pods, services, and persistent volume claims.

Implementing an External Monitoring Strategy

While OpenShift includes built-in monitoring, relying solely on internal tools is often insufficient for enterprise-grade requirements. Logs should be centralized, easily accessible, and—most importantly—independent from the objects being monitored. An external monitoring solution, such as the Elastic Stack, ensures that if the OpenShift cluster itself experiences a failure, the monitoring data remains available for analysis.

Since OpenShift is based on Kubernetes, the standard Kubernetes integration provided by Elastic can be utilized. The process involves:

1. Navigating to the integration section and searching for the Kubernetes integration.
2. Creating an agent policy that includes the Kubernetes integration and any additional required policies.
3. Generating the agent manifest via the Fleet > Add Agent section.
4. Customizing the manifest for OpenShift compatibility, specifically by creating a secret to store the fleet enrollment token to ensure security.

Data Ingestion and Observability Configuration

To transform the Elastic Stack from a set of installed tools into a functioning monitoring system, data must be shipped from the OpenShift nodes to Elasticsearch. This is achieved using Beats.

The Role of Beats

Beats are lightweight shippers installed on target devices to feed information to Logstash or Elasticsearch. There are several types of Beats utilized in this architecture:

• Filebeat: Specifically designed for logs and files.
• Metricbeat: Used for gathering system and service statistics.
• Packetbeat: Used for analyzing network packets.
• Winlogbeat: Used for collecting Windows event logs.

In an OpenShift environment, Filebeat and Metricbeat are the primary tools used to ingest observability data.

Step-by-Step Deployment of Beats on OpenShift

The deployment of Beats requires a specific sequence of commands to ensure the shippers have the necessary permissions and network access to function within the OpenShift security model.

1. Download the manifest files:
   curl -L -O https://raw.githubusercontent.com/liladler/eck-on-openshift/main/filebeat_openshift.yaml
curl -L -O https://raw.githubusercontent.com/liladler/eck-on-openshift/main/metricbeat_openshift.yaml

2. Grant privileged permissions to the Beats service accounts to allow them to access node-level data:
   oc adm policy add-scc-to-user privileged system:serviceaccount:openshift-monitoring:metricbeat
oc adm policy add-scc-to-user privileged system:serviceaccount:openshift-monitoring:filebeat

3. Patch the node-selector for the openshift-monitoring namespace to ensure the pods can be scheduled on any available node:
   oc patch namespace openshift-monitoring -p '{"metadata": {"annotations": {"openshift.io/node-selector": ""}}}'

4. Enable network communication between the elastic project and the openshift-monitoring project:
   oc adm pod-network join-projects --to=elastic openshift-monitoring

5. Apply the final manifests to deploy the shippers:
   oc apply -f filebeat_openshift.yaml
oc apply -f metricbeat_openshift.yaml

Component Summary and Specification

The following table provides a technical overview of the components involved in the OpenShift Elastic Stack deployment.

Comprehensive Analysis of System Monitoring

IT system monitoring is a proactive operational discipline aimed at preventing outages and downtime by measuring current system behavior against predetermined baselines. In a complex environment like OpenShift, this involves tracking several critical metrics:

• CPU Usage: Identifying bottlenecks or runaway processes.
• Memory Usage: Detecting leaks and ensuring proper resource allocation.
• Network Traffic: Monitoring throughput across routers and switches to identify congestion.
• Application Performance: Conducting root-cause analysis for latency or errors.

Traditional methods, such as manual scripting or cron jobs that send email alerts upon baseline deviation, are insufficient for containerized environments. The ELK Stack replaces these fragmented methods with a centralized system. By utilizing the "Deep Drilling" approach to observability, an administrator can move from a high-level Kibana dashboard (showing a spike in CPU) down to the specific Filebeat logs from a single pod in the OpenShift cluster, all within a matter of seconds. This integration transforms the monitoring process from a reactive "firefighting" exercise into a proactive stability strategy.

Conclusion

The deployment of the Elastic Stack on Red Hat OpenShift via the ECK Operator represents a sophisticated approach to infrastructure observability. By shifting away from fragmented, script-based monitoring toward a centralized, operator-managed architecture, organizations can achieve a level of visibility that is critical for managing the complexities of modern containerization. The synergy between Elasticsearch's indexing power, Logstash's transformation capabilities, and Kibana's visualization tools—all streamlined by the ECK Operator—allows for a seamless transition from day-one installation to day-two operations. The rigorous requirement for kernel-level adjustments, such as increasing vm.max_map_count, and the precise configuration of Security Context Constraints (SCCs) for Beats, underscores the technical depth required to maintain a stable production environment. Ultimately, the integration of Filebeat and Metricbeat ensures that the telemetry pipeline is complete, providing the unified view necessary to maintain high availability and operational excellence in an OpenShift ecosystem.

Sources

1. How to Deploy the Elastic Stack on Red Hat OpenShift with ECK
2. How to monitor your OpenShift cluster with ElasticStack
3. What is ELK Stack