Integrating Kubernetes Logging Architectures via the ELK Stack and Filebeat

The implementation of a centralized logging system within a Kubernetes (k8s) environment is a critical requirement for maintaining the operational health of microservices. In a containerized ecosystem, logs are ephemeral; when a pod crashes or is rescheduled, the local logs residing within the container are lost. To solve this, the ELK Stack—comprising Elasticsearch, Logstash, and Kibana—is deployed to create a persistent, searchable, and scalable telemetry pipeline. This architecture ensures that every event, from a standard application stdout stream to a critical system error, is captured, indexed, and visualized in real-time.

The core objective of this integration is to transform raw, unstructured text streams from various pods into structured data that can be queried. By utilizing Filebeat as a lightweight shipper, the system avoids the resource overhead of running full Logstash instances on every node, instead leveraging a "shipper-processor-indexer-visualizer" chain. This ensures that the Kubernetes cluster remains performant while providing deep observability into the application layer.

The ELK Stack Component Architecture

The ELK stack is not a single application but a suite of three distinct technologies that operate in a pipeline to handle data-driven applications and log analytics.

Elasticsearch: The Search and Analytics Engine

Elasticsearch serves as the heart of the stack, functioning as a scalable search and analytics engine. It is essentially a distributed document store that allows for near real-time search of massive volumes of data.

• Nodes: These are the dedicated servers upon which Elasticsearch runs. Each node participates in the cluster, sharing the workload of search and analytics tasks.
• Shards: To ensure high availability and performance, the database space is logically divided into shards. This distribution allows Elasticsearch to parallelize search queries across multiple nodes, significantly increasing data accessibility.
• Indices: Data is organized into indices. An index is a collection of documents that share similar characteristics, facilitating efficient data management and retrieval.

The technical impact of this architecture is that it allows a Kubernetes cluster to store terabytes of logs without sacrificing query speed. Because Elasticsearch is designed for distributed environments, it aligns perfectly with the scalable nature of Kubernetes.

Logstash: The Data Processing Pipeline

Logstash acts as the processing bridge. It is a server-side tool designed to collect logs from various sources, parse them, and transmit them to Elasticsearch.

• Collection: Logstash can ingest data from multiple sources, including files, sockets, and message queues.
• Parsing: It transforms raw logs into structured formats. For example, a raw text single-line log can be parsed into specific fields such as timestamp, log level, and message.
• Transmission: Once processed, Logstash sends the structured data to Elasticsearch for final storage.

In a high-volume environment, Logstash ensures that the data hitting the database is clean and optimized, reducing the computational load on the Elasticsearch indexing process.

Kibana: The Visualization Layer

Kibana is the window into the data. It is a powerful visualization tool that interacts directly with Elasticsearch.

• Exploration: Users can use the Discover console to search through logs using KQL (Kibana Query Language).
• Visualization: It transforms raw logs into interactive charts, graphs, and dashboards.
• Monitoring: By creating specific data views, operators can monitor the health of their Kubernetes pods in real-time through visual alerts.

Filebeat Implementation and Log Collection

While Logstash is powerful, it is resource-intensive. Filebeat is deployed as a lightweight shipper to bridge the gap between the Kubernetes nodes and the ELK stack.

Log Source Identification in EKS

In Amazon Elastic Kubernetes Service (EKS), logs are typically stored in a specific directory on the worker nodes. The standard path is:

/var/log/containers

To verify the presence of logs, a technician must log into a node and navigate to this directory. If the directory is empty, it indicates a configuration issue with the container runtime or the node's logging driver. This directory is crucial because it contains the symlinks to the actual log files generated by the Docker or containerd runtime.

Filebeat Configuration and Deployment

Filebeat is configured to monitor the /var/log/containers path and ship those logs to the ELK pipeline. The deployment typically involves applying Kubernetes templates:

kubectl apply -f filebeat-k8s

The configuration requires specific settings to ensure the shipper can communicate with the cluster. The primary files needed for container initialization include the configuration map and the service account. The process often involves using a ConfigMap to define how Filebeat should read the logs and where it should send them.

The technical flow for Filebeat includes:

• Reading logs from the local node path.
• Adding Kubernetes metadata (pod name, namespace, labels) to the log entry.
• Shipping the enriched log via the Beats protocol to Logstash or directly to Elasticsearch.

Step-by-Step Deployment Workflow

Deploying the ELK stack requires a coordinated sequence of actions to ensure that the database is ready before the shippers begin sending data.

Phase 1: Infrastructure Setup

Before deploying the software, a Kubernetes cluster must be active. This can be achieved through a manual setup or by using a GitHub repository containing Terraform files to automate the provisioning of the cluster. For local testing, Minikube is frequently used to simulate the environment.

Phase 2: Elasticsearch and Kibana Installation

The most efficient way to deploy Elasticsearch is through Helm charts. Helm manages the complex dependencies and configurations required for a distributed database.

Once the pods are running, the administrator must retrieve the credentials for the master user to access the Kibana dashboard. This is done by extracting the secrets from the Kubernetes namespace:

kubectl get secret elasticsearch-master-credentials -o jsonpath="{.data.username}" | base64 --decode ; echo

kubectl get secret elasticsearch-master-credentials -o jsonpath="{.data.password}" | base64 --decode ; echo

Phase 3: Log Streaming and Application Deployment

To test the pipeline, a sample application must be deployed to generate logs. From the cloned repository manifests folder, the following command is executed:

kubectl apply -f app -n default

Once the application is live, the logs are streamed from the pod, picked up by Filebeat, processed, and indexed.

Phase 4: Kibana Index and Data View Configuration

After the installation is complete and the application is generating data, the user must configure Kibana to recognize the data:

• Navigate to the Discover console.
• Create an Elasticsearch index pattern.
• Create a data view.

This process tells Kibana which index in Elasticsearch it should look at to display the logs. If logs are not appearing, the operator should generate more requests to the application to trigger new log entries and troubleshoot the connection.

Technical Data Flow and Interconnections

The movement of data from a container to a visual graph involves several layers of transformation.

The Data Pipeline Sequence

The interaction between components follows a strict sequence:

• Client Stage: Containers produce logs in formats such as JSON, YAML, or CSV. JSON is the preferred format for seamless integration.
• Input Stage: Log receivers at the input nodes access the appropriate repository nodes.
• Processing Stage: Content handlers map the data downstream through ES processors.
• Output Stage: The data is finally aggregated into time-domain based nodes within Elasticsearch.

Logic and Routing Table

Advanced Configuration and Troubleshooting

Maintaining a scalable logging system requires fine-tuning of the configuration and a deep understanding of the failure points.

Persistent Storage and Data Integrity

For a production-grade ELK stack, persistent volumes are mandatory. Without them, if an Elasticsearch pod restarts, all indexed logs are deleted. The use of PersistentVolumeClaims (PVC) ensures that the data is stored on external disks (like AWS EBS or GCP Persistent Disks) rather than the ephemeral storage of the pod.

Troubleshooting the Log Stream

When logs fail to appear in Kibana, the following troubleshooting steps are required:

• Verify the Filebeat pod status: Ensure the shipper is not in a CrashLoopBackOff state.
• Check the log paths: Confirm that the /var/log/containers directory is correctly mounted as a volume in the Filebeat pod.
• Validate credentials: Ensure that the elasticsearch-master-credentials are correct and that the Filebeat output settings match the database password.
• Test connectivity: Use curl to check if the Elasticsearch API is reachable from within the cluster:
  curl -X GET "localhost:9200/_cat/indices?v"

Conclusion: Analysis of the Integrated Logging Solution

The integration of the ELK stack with Filebeat transforms Kubernetes logging from a manual, pod-by-pod inspection process into a centralized intelligence system. By decoupling the log collection (Filebeat) from the processing (Logstash) and storage (Elasticsearch), the architecture achieves a high degree of fault tolerance.

The primary value proposition of this setup is the ability to perform "cross-pod correlation." Because Filebeat attaches Kubernetes metadata to every log line, an operator can filter logs across an entire namespace or a specific label set, regardless of which physical node the pod was running on. This is indispensable for debugging microservices where a single user request may traverse five different services.

Furthermore, the use of Elasticsearch indices allows for the implementation of data retention policies, ensuring that old logs are archived or deleted to save costs, while recent logs remain instantly searchable. The end result is a robust, observable environment where the time to detect an issue (MTTD) and the time to resolve it (MTTR) are significantly reduced through data-driven insights.

Sources

1. IT Syndicate
2. CodeZup
3. FossTechNix