The deployment of the Elastic Stack within a Kubernetes environment represents a critical intersection of distributed search and analytics and modern container orchestration. At its core, the Elastic Stack—comprising Elasticsearch, Logstash, and Kibana—provides a robust framework for searching, analyzing, and visualizing data in real-time. When these components are deployed via Helm, the Kubernetes package manager, the complexity of managing stateful sets, service accounts, and complex configuration files is abstracted into a streamlined, repeatable process.
The transition toward using Helm charts for the Elastic Stack is specifically designed to facilitate a lightweight method of configuring official Docker images provided by Elastic. However, the architectural paradigm has shifted toward the Elastic Cloud on Kubernetes (ECK) Operator. The ECK Operator serves as a Kubernetes operator that manages the lifecycle of Elastic Stack resources, providing higher-level abstractions than standard Helm charts alone. By leveraging the eck-stack Helm chart, administrators can deploy core components such as Elasticsearch and Kibana together under a single release, ensuring that the underlying infrastructure is managed by the operator for increased stability and operational efficiency.
The Architectural Foundation of the Elastic Stack
The Elastic Stack is not a monolithic entity but a collection of specialized tools that work in synergy to handle the entirety of the data pipeline, from ingestion to visualization.
Elasticsearch: The Distributed Core
Elasticsearch serves as the central engine of the stack. It is a distributed, RESTful search and analytics engine responsible for storing and indexing log data.
Technically, Elasticsearch utilizes a distributed document store that allows for horizontal scaling. In a Kubernetes environment, this means the system can dynamically spin up new nodes to handle increased data loads. The impact for the user is a system that maintains high availability and performance even as the volume of ingested logs grows into the terabytes. This connects directly to the Helm deployment process, where the eck-elasticsearch chart defines the resource requirements and storage classes necessary for the database to persist data across pod restarts.
Kibana: The Visualization Layer
Kibana acts as the window into the data stored within Elasticsearch. It provides a graphical user interface (GUI) for querying the data and creating visual dashboards.
From a technical perspective, Kibana is a client-side application that communicates with Elasticsearch via its API. For users, this means they do not need to write complex Query DSL (Domain Specific Language) to analyze logs; they can use the Kibana interface to filter and visualize trends. Within the Helm ecosystem, the eck-kibana chart ensures that the visualization layer is correctly networked to reach the Elasticsearch cluster, typically within the same Kubernetes namespace.
Logstash: The Processing Pipeline
While Elasticsearch and Kibana form the core, Logstash provides an optional but powerful layer for log processing. It is designed for scenarios where specific log processing requirements exist, such as parsing structured logs or applying complex filters before the data reaches Elasticsearch.
The technical implementation of Logstash involves the creation of input, filter, and output stages. For the end-user, this means that raw, messy log files can be transformed into clean, structured JSON documents. In the context of Helm, Logstash is often deployed as a separate chart (elastic/logstash), allowing it to be scaled independently of the database layer to handle spikes in log ingestion.
Comprehensive Helm Installation Workflow
Deploying the Elastic Stack requires a systematic approach to ensure that the environment is prepared and the repositories are correctly synchronized.
Prerequisites and Environment Setup
Before initiating the installation, the operator must ensure that Helm is installed on the local operating system. The minimum supported version of Helm for these deployments is 3.2.0.
The process begins with the creation of a dedicated Kubernetes namespace to ensure logical isolation of the monitoring stack. For example, creating a namespace named monit or elastic-stack prevents resource collisions with other applications in the cluster.
The command to create a namespace is:
kubectl create namespace monit
Repository Management
To access the official Elastic charts, the Elastic Helm repository must be added to the local Helm configuration. This allows the Helm client to fetch the latest chart definitions directly from Elastic's servers.
The following sequence of commands is required to initialize the repository:
helm repo add elastic https://helm.elastic.co
helm repo update
The helm repo add command registers the URL, and helm repo update ensures the local cache is synchronized with the latest versions available in the remote repository. This ensures that the user is deploying the most recent and secure versions of the software.
Deploying the Elastic Stack via ECK
Starting from ECK version 2.4.0, a specialized Helm chart (eck-stack) is available. This chart is an umbrella wrapper built on top of individual charts like eck-elasticsearch and eck-kibana.
The primary benefit of using the eck-stack chart is the ability to deploy the core components together under a single release. This simplifies the lifecycle management of the stack, as a single helm upgrade command can potentially update the entire environment.
Version Management and Upgrade Strategies
A critical aspect of maintaining a production-grade Elastic Stack is the ability to control versioning. By default, a Helm chart version defines a set of default Elastic Stack component versions. However, professional environments often require explicit version pinning.
Explicit Version Overrides
If an administrator needs to use a specific version (such as version 9.3.3) that differs from the chart's default, they can override the component versions using either the --set flag during installation or through a values.yaml file.
Using the --set parameter allows for rapid, command-line driven updates. For an existing release named es-kb-quickstart, the upgrade command would be:
helm repo update
helm upgrade es-kb-quickstart elastic/eck-stack -n elastic-stack --set eck-elasticsearch.version=9.3.3 --set eck-kibana.version=9.3.3
This technical approach ensures that components without an explicitly defined version continue to use the chart defaults, while critical components are locked to a known stable version.
Configuration via Values Files
For more sustainable management, especially in CI/CD pipelines, a custom-values.yaml file is recommended. This file acts as the single source of truth for the deployment configuration.
The following structure must be used in the values file to specify versions:
yaml
eck-elasticsearch:
version: 9.3.3
eck-kibana:
version: 9.3.3
By referencing this file during a helm upgrade or helm install, the operator ensures consistency across different environments (Development, Staging, Production).
Upgrading the Helm Release
To upgrade an installed release to the latest version of the Helm chart itself, the following process is used:
helm repo update
helm upgrade es-kb-quickstart elastic/eck-stack -n elastic-stack
Technically, this process refreshes the local chart cache and then applies the new chart version. By default, this also upgrades the Elastic Stack components to the versions associated with that specific chart version, unless explicit overrides are in place.
Advanced Component Integration
Beyond the core deployment, the Elastic Stack can be extended with additional agents and specialized processing tools.
Integration with AutoOps
For users requiring advanced operational intelligence, the Elastic Stack can be connected to AutoOps. This is achieved by installing the eck-stack with specific custom values that trigger the creation of an AutoOpsAgentPolicy resource.
The command to implement this connection is:
helm install eck-stack-with-autoops elastic/eck-stack --values https://raw.githubusercontent.com/elastic/cloud-on-k8s/3.3/deploy/eck-stack/examples/autoops/basic.yaml -n elastic-stack
This integration allows the Elasticsearch clusters to communicate with AutoOps, providing enhanced monitoring and automated operational insights.
Logstash Deployment and Verification
When the standard ingestion methods are insufficient, Logstash is deployed as a standalone component within the monitoring namespace.
The installation command is:
helm install logstash elastic/logstash --namespace monit
Once the installation is triggered, the operator must verify the status of the pods to ensure the container has reached a "Ready" state. This is done using the following command:
kubectl get pods --namespace=monit -l app=logstash-logstash -w
The expected output indicates that the pod logstash-logstash-0 is in the Running state with 1/1 containers ready.
Operational Post-Installation Tasks
Installation is only the first step. To make the Elastic Stack functional, several configuration tasks must be performed.
Kibana Index Pattern Configuration
Before data can be visualized, Kibana must be told how to interpret the data stored in Elasticsearch. This is achieved by configuring Index Patterns.
Technically, an index pattern defines which Elasticsearch indices Kibana should query. Without this, the visualization tools cannot "see" the data. This is a mandatory step for any user wishing to leverage Kibana's search and dashboard capabilities.
Logstash Pipeline Configuration
If Logstash was installed, it remains a blank slate until pipelines are configured. These pipelines define how logs are ingested and processed.
Configuration can be managed by:
- Defining Logstash configuration files.
- Mounting these files as Kubernetes ConfigMaps.
- Utilizing other external configuration management tools.
Comprehensive Observability
To ensure the health of the stack, monitoring must be implemented for both Elasticsearch and Kibana. This can be achieved through:
- Built-in Elastic metrics.
- Integration with external solutions like Prometheus and Grafana.
This creates a secondary monitoring layer that alerts administrators if the primary monitoring stack (ELK) experiences downtime.
Strategic Analysis of ECK vs. Standard Helm Charts
There is a significant distinction between using lightweight Helm charts and using the Elastic Cloud on Kubernetes (ECK) Operator.
The ECK Advantage
Elastic recommends ECK as the primary method for running the Elastic Stack on Kubernetes. The operational benefits are substantial:
- Automated Recovery: ECK can automatically spin up cluster nodes that were lost due to failed underlying infrastructure.
- Seamless Upgrades: The operator handles the complexities of rolling upgrades, ensuring that the cluster remains available while versions are bumped.
- Rolling Changes: Configuration changes can be rolled out across the cluster without causing total downtime.
The Transition of Community Helm Charts
As of the release of Elastic version 8.5.1, Elastic shifted the maintenance of the general Elastic Stack Helm charts to the community and contributors. This means that while these charts are still supported under End-of-Life (EOL) limitations, the primary development focus has moved to the Helm charts that specifically apply to ECK Custom Resources.
This shift indicates a move toward "Operator-based" management, where the Helm chart serves as the initial bootstrapper for the Operator, and the Operator subsequently manages the lifecycle of the actual Elastic resources.
Technical Specifications Summary
The following table provides a structured overview of the deployment requirements and components.
| Component | Role | Helm Chart Name | Min. Helm Version | Management Method |
|---|---|---|---|---|
| Elasticsearch | Storage & Indexing | eck-elasticsearch |
3.2.0 | ECK Operator |
| Kibana | Visualization | eck-kibana |
3.2.0 | ECK Operator |
| Logstash | Log Processing | elastic/logstash |
3.2.0 | Standard Helm |
| Elastic Stack | Umbrella Deployment | eck-stack |
3.2.0 | ECK Operator |
Conclusion
The deployment of the Elastic Stack via Helm on Kubernetes is a sophisticated process that balances the ease of package management with the power of the ECK Operator. By utilizing the eck-stack chart, administrators can ensure that their search and analytics infrastructure is not only easily deployable but also resilient and scalable.
The transition from simple chart-based deployments to operator-managed resources represents a maturation of the Elastic ecosystem, providing critical features like self-healing nodes and seamless rolling upgrades. For the end-user, the primary focus must remain on the precise control of component versions and the careful configuration of Logstash pipelines and Kibana index patterns. Ultimately, the synergy between Helm's deployment capabilities and ECK's operational management creates a production-ready environment capable of handling massive data volumes with minimal administrative overhead.