Centralized Observability Architecture on OpenShift Container Platform

The transition from monolithic application architectures to distributed microservice ecosystems fundamentally alters the operational requirements for system observability, debugging, and performance monitoring. In a traditional monolithic deployment, engineers could attach debuggers directly to a single running process, set breakpoints, and trace execution paths with relative simplicity. Modern distributed environments, however, scatter functionality across hundreds or thousands of containerized instances, load-balanced network endpoints, and ephemeral pod lifecycles. Attaching a debugger to a specific process in such an environment becomes operationally unfeasible due to the dynamic nature of orchestration layers and the sheer volume of concurrent instances executing identical microservice logic. This architectural reality necessitates a centralized, cluster-level logging and telemetry stack capable of aggregating, processing, indexing, and visualizing data across the entire infrastructure. The Elastic Stack, historically referred to as the ELK Stack, provides the foundational components required to construct this observability pipeline within Red Hat OpenShift Container Platform environments. The stack operates through a coordinated workflow where lightweight data shippers collect telemetry, processing engines normalize and enrich the data, distributed search engines index and persist the information, and web-based dashboards enable analytical querying and visualization. Deploying this stack within an OpenShift cluster requires careful consideration of deployment methodologies, storage provisioning, network routing, security contexts, and operator-driven lifecycle management. The implementation pathways range from manual container orchestration using Docker and Source-to-Image workflows, to Ansible-driven infrastructure automation, to modern Kubernetes-native operator deployments via Elastic Cloud on Kubernetes. Each methodology carries distinct operational characteristics, scalability parameters, and administrative overhead requirements that directly influence production readiness, high availability guarantees, and long-term maintainability. Understanding the technical mechanics, architectural trade-offs, and configuration directives for each deployment model is essential for infrastructure engineers tasked with building resilient observability pipelines that can withstand production workloads, node failures, and scaling events without compromising data integrity or query performance.

Architectural Foundations and Data Flow

The foundational architecture of a centralized logging stack relies on a unidirectional data pipeline that moves telemetry from source applications through collection, processing, storage, and finally to visualization interfaces. In a standard development or small-scale deployment scenario, the data flow follows a direct linear progression where lightweight data shippers handle initial collection, a processing engine performs aggregation and transformation, a distributed search engine manages indexing and persistence, and a web interface provides analytical capabilities. This classic architecture can be represented as follows:

• Data collection agents ship raw logs and metrics from containerized workloads
• An aggregation and processing layer normalizes formats, filters noise, and enriches context
• A distributed search and storage layer indexes the processed data for rapid retrieval
• A visualization layer provides query interfaces and dashboard rendering capabilities

While this linear architecture suffices for low-volume environments, production deployments processing high-throughput telemetry require additional buffering mechanisms to prevent data loss during processing bottlenecks or downstream storage degradation. The introduction of a message broker or queueing system between the collection layer and the processing layer creates a decoupled architecture that absorbs traffic spikes, allows independent scaling of processing workers, and guarantees delivery through acknowledgment mechanisms. Production-scale architectures typically incorporate buffering technologies such as Kafka, RabbitMQ, or Redis to create a resilient pipeline that can handle backpressure without dropping telemetry data.

The necessity of this centralized architecture stems directly from the debugging challenges inherent in distributed systems. When a microservice instance fails or exhibits anomalous behavior, engineers cannot simply step through code in a local debugger. Instead, they must correlate log entries across multiple services, identify temporal patterns, and trace request IDs across network boundaries. A centralized logging stack transforms scattered, ephemeral log files into a unified, queryable corpus that enables temporal correlation, pattern matching, and root cause analysis. The technical implementation of this architecture requires precise configuration of data routing, index lifecycle management, and resource allocation to prevent the observability stack itself from becoming a performance bottleneck or a single point of failure.

Containerized Deployment and Source-to-Image Workflows

Before the advent of Kubernetes-native operators, deploying the Elastic Stack on OpenShift relied heavily on direct container management and Source-to-Image build pipelines. Modified container images often required adjustments to align with OpenShift's security constraints, particularly regarding process execution and file permissions. Standard upstream images frequently utilize privilege escalation utilities that conflict with OpenShift's restricted security contexts, necessitating the removal of such utilities and the enforcement of world-readable permissions on critical configuration files to ensure successful initialization.

Deploying these modified images manually involves a sequence of container lifecycle commands that establish the required services in isolation before networking them together. The initial step typically involves launching the search and storage component using a direct container execution command. Subsequent steps launch the visualization layer, explicitly linking it to the storage service and exposing the web interface port to the host network. The processing engine requires additional build steps when utilizing custom configuration repositories, leveraging the Source-to-Image tool to compile configuration files into a runnable container image.

• Launch the search and storage component using a direct container execution command
• Initialize the visualization layer with explicit service linking and port mapping
• Compile the processing engine configuration using the Source-to-Image build tool
• Execute the processing container with network linkage to the storage service

When transitioning from manual container commands to OpenShift-native application deployment, administrators utilize project isolation and application instantiation commands to establish the environment. Creating a dedicated project namespace isolates the observability stack from other cluster workloads, providing resource quotas and security boundaries. Application instantiation commands deploy each component as an OpenShift application, pulling from the modified image repositories. The processing engine requires additional parameters to specify the configuration repository and build context directory during deployment. Once deployed, the visualization component must be exposed to external networks through an OpenShift route, enabling browser-based access to the dashboard interface.

Ansible-Driven Provisioning and Cluster Sizing

Infrastructure-as-code methodologies provide a repeatable, version-controlled approach to deploying and scaling observability stacks across OpenShift clusters. Ansible playbooks serve as the primary automation mechanism for configuring cluster logging, with specific variables controlling node counts, replica distribution, and high availability parameters. The cluster size is determined during the initial installation phase through a dedicated variable that specifies the number of search and storage nodes. This variable establishes the baseline capacity of the indexing layer and influences shard allocation, routing tables, and inter-node communication overhead.

Scaling an existing cluster requires modifying the inventory configuration file and re-executing the logging playbook. This approach ensures that configuration changes are applied consistently across all nodes, preventing configuration drift that commonly occurs with manual adjustments. Additional clustering parameters can be modified through the same inventory mechanism, allowing administrators to tune performance characteristics, network timeouts, and garbage collection settings without redeploying the entire stack.

High availability in distributed search engines requires a minimum of three nodes distributed across separate physical or virtual hosts to prevent split-brain scenarios and ensure fault tolerance. The replica count variable determines how many copies of each data segment are maintained across the cluster. Setting the replica count to one ensures that two nodes hold a complete copy of all cluster data, providing immediate failover capability if a single node experiences hardware failure or network partitioning. Increasing the replica count to three results in four total copies of the data across the cluster, significantly enhancing durability but consuming additional storage resources and increasing synchronization overhead during write operations.

Viewing the current deployment state requires querying the cluster for specific deployment configurations using selector labels that identify logging infrastructure components. This query returns the status, replica count, and version information for each node in the cluster, enabling administrators to verify that scaling operations completed successfully and that all nodes are reporting healthy states.

oc get dc --selector logging-infra=elasticsearch

Storage Architecture and Host-Level Configuration

Persistent storage for distributed search engines requires careful consideration of performance characteristics, failure domains, and administrative overhead. OpenShift provides multiple storage options, including network-attached storage through NFS, host-path volumes, and local disk attachments. NFS-backed storage requires manual mounting procedures on each node, followed by ownership adjustments to ensure the container runtime can write data without violating security contexts.

The mounting procedure involves specifying the NFS server address, the remote share path, and the local mount point. After mounting, the ownership of the directory must be adjusted to match the user and group identifiers expected by the search engine container. Each node in the cluster requires a separate backing file or mount point to prevent data conflicts and ensure proper shard distribution across the storage layer. These mount operations must be maintained manually outside of the container orchestration layer, as containerized processes lack the privilege to modify host filesystem mounts.

Local disk volumes offer superior performance for indexing workloads due to reduced network latency and higher IOPS, but they require specific security context modifications to function within OpenShift's restricted environment. The relevant service account must be granted privileged access to mount and modify local volumes, bypassing standard security constraints that prevent containers from interacting with host block devices. This escalation is performed through administrative policy commands that grant the logging service account access to the privileged security context.

oc adm policy add-scc-to-user privileged \ system:serviceaccount:openshift-logging:aggregated-logging-elasticsearch

Administrators upgrading from earlier OpenShift versions must verify the project namespace where the logging stack was originally deployed, as naming conventions may have changed between major releases. Legacy deployments may reside in a namespace with a different identifier, requiring namespace-specific commands to query deployment status and modify configurations.

Security Contexts and External Route Configuration

External access to the search and storage layer enables third-party monitoring tools, custom analytics applications, and direct query interfaces to interact with the observability data. Enabling external access requires configuration variables that permit external routing and specify the public hostname that will be presented to clients. These variables are applied through the logging configuration playbook, which regenerates routes, certificates, and network policies to accommodate external traffic.

openshift_logging_es_allow_external=True
openshift_logging_es_hostname=elasticsearch.example.com

Executing the configuration playbook applies these changes to the cluster, updating route definitions and generating the necessary cryptographic material for secure communication. The playbook directory contains the automation logic that reconciles the desired state with the actual cluster configuration, ensuring that routing rules, certificate domains, and access controls are applied consistently.

cd /usr/share/ansible/openshift-ansible
ansible-playbook [-i </path/to/inventory>] \ playbooks/openshift-logging/config.yml

Remote authentication to the search engine requires specific HTTP headers that convey identity, authorization tokens, and client network origin. These headers must be included in every request to satisfy the proxy authentication layer that protects the internal service endpoints. The authorization header carries the authentication token, the remote user header identifies the authenticated principal, and the forwarded-for header preserves the originating IP address for audit logging and access control enforcement.

• Authorization: Bearer $token
• X-Proxy-Remote-User: $username
• X-Forwarded-For: $ip_address

Access control policies require that the requesting principal possesses project-level permissions within the namespace where the logging stack operates. Without appropriate role bindings or policy grants, authentication requests will be rejected regardless of valid token presentation. This enforcement ensures that observability data remains isolated within project boundaries and cannot be accessed by unauthorized workloads or users outside the designated namespace.

Elastic Cloud on Kubernetes and Operator Integration

The evolution of Kubernetes-native observability deployment has shifted toward operator-based management, culminating in the Elastic Cloud on Kubernetes offering. This operator provides an official, maintained pathway for deploying the Elastic Stack on OpenShift, abstracting away manual container configuration, Ansible playbook management, and storage provisioning complexities. Installation begins through the OpenShift OperatorHub interface, where administrators select the Elastic operator and initiate the installation process using default configuration selections.

Once installed, the operator deploys into a dedicated namespace reserved for cluster-wide operators, ensuring isolation from application workloads while maintaining access to cluster resources. Verifying operator health requires querying the namespace for pods matching the operator's control plane labels. This query returns the pod name, readiness status, restart count, and age, providing immediate visibility into the operator's operational state.

oc get pods -n openshift-operators -l control-plane=elastic-operator

Retrieving operator logs provides detailed insight into reconciliation cycles, resource creation events, and error conditions. The logs contain structured JSON entries that document each reconciliation run, including version information, namespace context, and component identifiers. Continuous log streaming enables administrators to monitor the operator's progress as it provisions Elasticsearch clusters, creates Kibana instances, and manages certificate generation for secure communication.

oc logs -l control-plane=elastic-operator -n openshift-operators -f

OpenShift environments include preinstalled monitoring components that collect platform metrics, node health data, and application performance indicators. The Elastic operator integrates with these existing components, enabling centralized collection of OpenShift telemetry within the Elasticsearch indexing layer. Deployment of a three-node Elasticsearch cluster through the operator provides the high availability foundation required for production workloads, with the operator automatically managing shard allocation, node discovery, and certificate rotation.

The operator's reconciliation loop continuously monitors the desired state defined in custom resource definitions and applies necessary changes to maintain configuration consistency. This automated lifecycle management eliminates manual intervention for routine scaling operations, version upgrades, and certificate renewals, significantly reducing operational overhead while improving deployment reliability.

Conclusion

The implementation of a centralized observability stack on OpenShift Container Platform represents a critical infrastructure investment that directly impacts operational efficiency, incident resolution speed, and system reliability. The architectural progression from linear data pipelines to buffered, high-availability clusters reflects the operational demands of modern distributed environments, where debugging ephemeral microservices requires correlated, queryable telemetry rather than isolated log files. Deployment methodologies have evolved from manual container orchestration and Source-to-Image build workflows to Ansible-driven infrastructure automation, and ultimately to Kubernetes-native operator management through Elastic Cloud on Kubernetes. Each approach offers distinct advantages in terms of control granularity, automation level, and administrative overhead.

Storage configuration remains a foundational constraint that dictates indexing performance, data durability, and maintenance complexity. Network-attached storage provides flexibility but requires manual mount management and ownership configuration, while local disk volumes offer superior IOPS at the cost of security context escalation and host-level administration. External access configuration introduces additional security considerations, requiring precise HTTP header authentication, project-level permission enforcement, and route certificate management to prevent unauthorized data exposure.

The adoption of operator-driven deployment fundamentally shifts the operational model from manual configuration management to declarative state reconciliation. The Elastic Cloud on Kubernetes operator automates cluster provisioning, replica management, certificate rotation, and version upgrades, reducing the risk of configuration drift and human error. This abstraction layer enables infrastructure teams to focus on observability strategy, data retention policies, and dashboard design rather than low-level container orchestration and storage provisioning. Organizations implementing centralized logging on OpenShift must carefully evaluate their throughput requirements, compliance mandates, and operational maturity to select the deployment methodology that aligns with their technical capabilities and long-term maintenance capacity. The transition to operator-managed observability stacks represents a maturation of platform engineering practices, establishing a foundation for automated, resilient, and scalable telemetry infrastructure that supports complex distributed workloads without compromising security or performance.

Sources

1. trimino.com simple-app kibana-elasticsearch-logstash-filebeat-openshift (https://trimino.com/simple-app/kibana-elasticsearch-logstash-filebeat-openshift/)
2. karkul openshift3-elk (https://github.com/karkul/openshift3-elk)
3. Red Hat OpenShift Container Platform 3.11 Configuring Clusters Install Config Aggregate Logging (https://docs.redhat.com/en/documentation/openshiftcontainerplatform/3.11/html/configuring_clusters/install-config-aggregate-logging)
4. Red Hat Run Elastic Cloud on Kubernetes on Red Hat OpenShift (https://www.redhat.com/en/blog/run-elastic-cloud-on-kubernetes-on-red-hat-openshift)