The deployment of the Elastic Stack—comprising Elasticsearch, Logstash, and Kibana—within a Kubernetes environment represents a sophisticated intersection of distributed systems and cloud-native orchestration. By leveraging Helm, the package manager for Kubernetes, organizations can abstract the complexity of YAML manifests into manageable charts, allowing for repeatable, scalable, and version-controlled deployments of the ELK ecosystem. This architectural approach transforms the manual labor of configuring individual pods and services into a streamlined process of value-driven configuration, ensuring that the core components of the logging pipeline are deployed with consistency across different environments, from development to production.
Infrastructure Foundations and GKE Environment Setup
The successful deployment of an ELK stack requires a robust underlying infrastructure capable of handling the high I/O and memory demands of Elasticsearch. When utilizing Google Kubernetes Engine (GKE), the process begins with the definition of the environment and the provisioning of the cluster.
The initial phase involves establishing global environment variables to ensure consistency across the deployment pipeline. For instance, defining the geographical location and the cluster identifier is critical for resource mapping:
$ LOCATION=us-central1-a
$ CLUSTER_NAME=kubetest
A critical technical consideration during the GKE setup is the selection of the machine type. By default, GKE often provisions nodes using the e2-medium machine type. From a technical specification standpoint, the e2-medium provides 2 vCPUs and 4 GB of memory. However, the Elastic Stack, particularly Elasticsearch, is extremely memory-intensive due to the Java Virtual Machine (JVM) heap requirements and the need for filesystem caching. Using an e2-medium instance is insufficient for a production-grade ELK deployment. The operational impact of utilizing underpowered nodes is immediate: the Kubernetes scheduler may trigger Out-Of-Memory (OOM) kills on the Elasticsearch pods, leading to a crash loop back-off state. Consequently, administrators must select machine types with significantly higher memory allocations to ensure cluster stability and optimal indexing performance.
Helm Repository Integration and Namespace Management
Helm acts as the abstraction layer that allows users to deploy complex applications via "charts." To deploy the Elastic Stack, the official Elastic Helm repository must be integrated into the local Helm environment.
The process of adding the repository is executed via the following commands:
$ helm repo add elastic https://helm.elastic.co
$ helm repo update
Adding the repository allows the Helm client to fetch the latest chart definitions, which include the necessary templates for Kubernetes services, deployments, stateful sets, and config-maps. The helm repo update command is an essential step that ensures the local cache is synchronized with the remote repository, preventing the installation of outdated or incompatible chart versions.
Once the repository is configured, the architectural best practice is to isolate the ELK stack within its own Kubernetes namespace. This ensures a logical separation of resources, simplifies access control (RBAC), and prevents naming collisions with other applications in the cluster.
$ kubectl create namespace monit
By creating a namespace such as monit or elastic-stack, the administrator defines a boundary for all ELK-related resources. This allows for granular monitoring of resource usage and simplifies the cleanup process if the stack needs to be redeployed.
Elasticsearch: The Core Indexing Engine
Elasticsearch serves as the heart of the ELK stack. Technically, it is a distributed, RESTful search and analytics engine that is responsible for storing, indexing, and searching log data. In a Kubernetes environment, Elasticsearch is typically deployed as a StatefulSet to ensure that each node maintains a persistent identity and that data is not lost during pod restarts.
The deployment of Elasticsearch requires a sophisticated handling of security credentials. Upon installation via Helm, the system generates master credentials that are stored as Kubernetes secrets. To access the administrative account for the cluster, these secrets must be retrieved and decoded from base64 format.
To retrieve the username:
kubectl get secrets --namespace=monit elasticsearch-master-credentials -ojsonpath='{.data.username}' | base64 -d
To retrieve the password:
$ kubectl get secrets --namespace=monit elasticsearch-master-credentials -ojsonpath='{.data.password}' | base64 -d
The use of ojsonpath allows the administrator to target the specific data field within the secret, while the base64 -d command converts the encoded string into a human-readable format. This security mechanism ensures that sensitive credentials are not stored in plain text within the cluster configuration.
Kibana: The Visualization and Dashboarding Layer
Kibana serves as the user interface for the ELK stack, providing the visualization and dashboarding component. It communicates with Elasticsearch via a REST API to query data and present it in a graphical format.
The technical integration of Kibana involves configuring it to point to the Elasticsearch service. When deployed via Helm, Kibana is typically managed as a deployment that includes a service of type ClusterIP or LoadBalancer to allow external access. The impact of a correctly configured Kibana instance is the ability for DevOps engineers to create real-time monitoring dashboards, visualize log trends, and perform deep-dive forensics into system errors without needing to write raw Query DSL.
The Transition to Elastic Cloud on Kubernetes (ECK)
A pivotal shift in the deployment strategy of the Elastic Stack is the move toward the Elastic Cloud on Kubernetes (ECK) operator. While standard Helm charts provide a lightweight way to configure official Docker images, they have limitations in operational flexibility.
Starting with ECK version 2.4.0, a dedicated Helm chart was introduced to manage Elastic Stack resources through the ECK Operator. This operator-based approach is superior to standard Helm deployments because it provides:
- Automated recovery: The operator can automatically spin up cluster nodes that were lost due to infrastructure failures.
- Seamless upgrades: The operator manages the rolling update of the stack, ensuring that the cluster remains available during version transitions.
- Rolling changes: Modifications to the cluster configuration can be applied incrementally without causing total downtime.
The eck-stack Helm chart is an umbrella chart built on top of individual specialized charts, such as eck-elasticsearch and eck-kibana. This modularity allows administrators to deploy core components together or independently depending on the architectural needs.
The following table illustrates the comparison between standard Helm deployments and the ECK Operator approach:
| Feature | Standard Helm Charts | ECK Operator |
|---|---|---|
| Management Style | Imperative/Template-based | Declarative/Operator-based |
| Recovery | Manual/K8s Restart | Automated Node Recovery |
| Upgrade Path | Manual Chart Upgrade | Seamless Rolling Upgrades |
| Complexity | Low to Medium | Medium (requires Operator) |
| Recommendation | Community Support | Official Elastic Recommendation |
Maintenance and Version Alignment
With the release of Elastic version 8.5.1, the maintenance of the standard Elastic Stack Helm charts transitioned toward the community and contributors. The official repository for these charts was slated for archiving six months after that release. Despite this, deployments made via these charts remain supported within the standard End-of-Life (EOL) limitations.
For organizations prioritizing stability, it is imperative that the Helm chart version aligns strictly with the version of the product being deployed. If a specific chart release exists for a given stack version, using that exact alignment ensures that the deployment has been tested against the corresponding production version of the software. This prevents version mismatch errors, where the Kubernetes manifest might request a feature or configuration option that is not supported by the underlying Docker image version.
Deployment Workflow and Command Sequence
The operational flow for deploying the stack using the Elastic Helm repository follows a specific sequence of commands to ensure that the environment is prepared before the resources are instantiated.
The sequence is as follows:
Initialize the Helm repository:
helm repo add elastic https://helm.elastic.coRefresh the local chart cache:
helm repo updateCreate the isolated environment:
kubectl create namespace elastic-stackDeploy the operator or the stack:
The specific command varies based on whether theeck-stackor a standalone chart is used, but it generally follows the pattern ofhelm install [release-name] elastic/eck-stack --namespace elastic-stack.
The minimum supported version of Helm for these operations is 3.2.0. Using an older version of Helm may result in template rendering failures or an inability to parse the latest chart specifications.
Conclusion
The deployment of the ELK stack on Kubernetes using Helm represents a transition from manual configuration to an automated, scalable operational model. While standard Helm charts provide an accessible entry point for deploying Elasticsearch, Logstash, and Kibana, the industry trend has shifted toward the Elastic Cloud on Kubernetes (ECK) operator. The operator-driven model solves the inherent challenges of managing stateful applications in a containerized environment by providing automated recovery and seamless upgrade paths.
From a technical perspective, the most critical factors for a successful deployment are the allocation of sufficient memory (moving beyond the e2-medium default in GKE), the strict alignment of chart versions with product versions, and the secure management of secrets via Kubernetes native mechanisms. By moving away from generic manifests and toward a combination of the ECK operator and tailored Helm charts, organizations can achieve a high-availability logging infrastructure that is resilient to infrastructure failure and easy to maintain throughout its lifecycle.