Metricbeat Kubernetes DaemonSet Architecture and Deployment

Metricbeat functions as a lightweight, high-efficiency telemetry agent designed specifically to provide comprehensive monitoring for Elasticsearch clusters and the underlying infrastructure of Kubernetes. By leveraging the Elasticsearch API, Metricbeat collects critical performance data and ships it directly back to an Elasticsearch instance for aggregation and analysis. When integrated into a Kubernetes environment, Metricbeat is typically deployed as a DaemonSet, ensuring that a dedicated monitoring agent resides on every single node within the cluster. This architectural choice ensures that the system respects the cluster topology while maintaining strict resource constraints, thereby granting administrators deep visibility into cluster health, capacity, and performance without introducing significant computational overhead.

The fundamental architecture of Metricbeat is based on a modular system. These modules are specialized components that collect metrics from specific services. For instance, the Elasticsearch module specifically queries the cluster API endpoints to retrieve a wide array of data, including node statistics, the overall cluster state, index-level metrics, and information regarding shard allocation. These collection cycles are executed on a configurable schedule, which is typically set to occur every 10 to 30 seconds. This high-frequency polling ensures that transient performance spikes are captured and that the operator has a near-real-time view of the system's operational status.

In the context of Kubernetes, the connectivity between Metricbeat and Elasticsearch is a critical configuration point. Metricbeat must be able to reach the Elasticsearch endpoints from within the cluster network. This necessitates the implementation of robust service discovery mechanisms, precise authentication configurations, and the application of network policies that allow the traffic to flow. Once collected, these metrics are transmitted to Elasticsearch, where they are stored in dedicated monitoring indices. These indices are then utilized by Kibana to populate pre-built dashboards, transforming raw metrics into visual insights. This self-monitoring pattern—where Elasticsearch is monitored by Metricbeat and visualized via Kibana—is highly efficient because the volume of monitoring metrics is generally manageable and does not overwhelm the cluster it is intended to monitor.

Kubernetes Deployment Strategies and Implementation

Implementing Metricbeat on Kubernetes can be approached through several methodologies, depending on the required level of automation and the existing tooling within the environment.

One primary method involves the use of Helm, which is often cited as the quickest installation path. By utilizing a Helm chart and a corresponding values file, administrators can deploy Metricbeat with minimal manual intervention. The process involves copying the Metricbeat chart's values file, adapting the configurations to match the specific environment, and executing a deployment command.

The following command illustrates the Helm installation process:

helm upgrade --install --force metricbeat stable/metricbeat --namespace metricbeat --values ./values.yaml

Beyond Helm, Metricbeat can be deployed using standard Kubernetes manifest files. The official manifests are designed to deploy Metricbeat as a DaemonSet. This ensures that one instance of the agent runs on every node, allowing for the retrieval of host-level metrics such as system CPU, memory, and disk usage, as well as Docker statistics and metrics from other services running atop the Kubernetes platform.

For users employing Kubernetes 1.7 or earlier, specific attention must be paid to data persistence. In these older versions, Metricbeat utilizes a hostPath volume located at /var/lib/metricbeat-data to persist internal data. The manifests for these versions utilize the DirectoryOrCreate folder autocreation feature, which was officially introduced in Kubernetes 1.8.

To retrieve the latest deployment manifests for version 8.19, the following command is used:

curl -L -O https://raw.githubusercontent.com/elastic/beats/8.19/deploy/kubernetes/metricbeat-kubernetes.yaml

DaemonSet Architecture and Leader Election

The deployment of Metricbeat as a DaemonSet is central to its effectiveness in a Kubernetes environment. Because a pod is scheduled on every node, the system can collect granular, node-specific data. However, some metrics are not node-specific but are instead cluster-wide. Examples of such data include Kubernetes events and kube-state-metrics. To prevent the duplication of this cluster-wide data—which would occur if every node-level agent reported the same global metrics—Metricbeat employs a leader election strategy.

Within the DaemonSet, one pod is designated as the leader. This leader pod holds a lock and assumes the responsibility for handling cluster-wide monitoring. The leader election configuration is managed via Autodiscover. In very large Kubernetes clusters, the unique setting in the configuration can be set to false. Doing so disables the leader election strategy in favor of running a dedicated Metricbeat instance via a separate Deployment in addition to the DaemonSet. This prevents the leader election process from becoming a bottleneck or a point of failure in massive environments.

By default, these components are deployed under the kube-system namespace. However, this can be modified within the manifest file to suit the organizational requirements of the cluster.

Detailed Configuration and Manifest Analysis

The configuration of Metricbeat in Kubernetes is managed through ConfigMaps and Jobs. This separation ensures that the setup process is handled independently of the continuous monitoring process.

The metricbeat-setup-config ConfigMap is used to define the initial configuration. This includes the path to module configurations, the index template settings, and the authentication credentials for the Elastic Cloud.

The following table outlines the key configuration parameters found in the setup ConfigMap:

To apply these configurations and initialize the dashboards in Kibana, a Kubernetes Job named metricbeat-setup is executed. This job runs a container with the image docker.elastic.co/beats/metricbeat:7.6.1 and passes the -e setup argument. This process ensures that the index templates and dashboards are loaded into the target Elasticsearch and Kibana instances before the DaemonSet begins shipping data.

The metricbeat-setup Job incorporates several critical environment variables and volume mounts:

• NODE_NAME: Derived from spec.nodeName to identify the node.
• CLOUD_ID: Retrieved from a secret named dynamic-logging.
• ELASTIC_PASSWORD: Retrieved from a secret named dynamic-logging.

The container in the setup job also requires specific security contexts, running as user 0 (root), and mounts the Docker socket via a hostPath volume at /var/run/docker.sock.

Metricbeat Modules and Kubernetes Integration

The metricbeat-daemonset-config provides the operational logic for the monitoring agents. A primary focus is the Kubernetes module, which allows Metricbeat to interface with the Kubernetes API and other metric providers.

The autodiscover provider for Kubernetes is configured to monitor the cluster scope. The configuration allows for the collection of a vast array of state-related metrics via kube-state-metrics. These metrics are retrieved from kube-state-metrics:8080 every 10 seconds.

The following lists the specific metricsets enabled for the Kubernetes state monitoring:

• state_namespace
• state_node
• state_deployment
• state_daemonset
• state_replicaset
• state_pod
• state_container
• state_job
• state_cronjob
• state_resourcequota
• state_statefulset
• state_service
• state_persistentvolume
• state_persistentvolumeclaim
• state_storageclass

In addition to state metrics, the Kubernetes module collects data directly from the API server. This is achieved by connecting to https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}. To authenticate these requests, Metricbeat uses a bearer token located at /var/run/secrets/kubernetes.io/serviceaccount/token and validates the connection using the service CA certificate at /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt.

Manual Installation and Troubleshooting

For those not using Helm or full Kubernetes manifests, a manual installation on a Kubernetes node is possible, although it is less scalable. This approach involves downloading a specific binary, such as metricbeat-6.5.4-1.x86_64.

During manual configuration, the operator must define the Elasticsearch and Kibana URLs. In a Kubernetes environment, these are often discovered using kubectl commands to find the service names.

The following shell commands are used to dynamically resolve the URLs:

[ "$RELEASE" == "" ] && RELEASE=metricbeat-v1

[ "$ELASTICSEARCH_URL" == "" ] && ELASTICSEARCH_URL=http://$(kubectl get svc -n elasticsearch | grep 'elastic.*client' | head -n 1 | awk '{print $1}'):9200

[ "$KIBANA_URL" == "" ] && KIBANA_URL=http://$(kubectl get svc -n elasticsearch | grep 'kibana' | head -n 1 | awk '{print $1}'):5601

Once the environment variables are set, the /etc/metricbeat/metricbeat.yml file is modified. To initialize the dashboards in Kibana, the following command is issued:

sudo metricbeat setup

This command loads the index templates and approximately 20 pre-built dashboards. To start the agent, the following service commands are utilized:

sudo service metricbeat start

To ensure the agent survives a system reboot, it must be enabled via systemd:

sudo systemctl enable metricbeat

If the Kubernetes module is not required during a specific test, it can be disabled using:

sudo metricbeat modules disable kubernetes

Once the service is active, the "System Overview" and "Host Overview" dashboards in Kibana will begin to populate with data, providing a high-level view of the hardware performance of the Kubernetes nodes.

Analysis of Monitoring Efficiency and Resource Impact

The integration of Metricbeat within a Kubernetes environment represents a strategic balance between visibility and overhead. By utilizing a DaemonSet, the architecture ensures that no node is left unmonitored, which is critical for detecting "noisy neighbor" effects or node-level resource exhaustion that could trigger pod evictions.

The impact of this deployment is minimized through several mechanisms. First, the use of the Elasticsearch API for collection is more efficient than polling individual pod logs or using heavy agents. Second, the 10-30 second collection interval prevents the monitoring system from becoming a source of network congestion. Third, the leader election strategy prevents the redundant transmission of cluster-wide metrics, which would otherwise lead to an exponential increase in data volume as the cluster grows.

The reliance on kube-state-metrics is a key efficiency driver. By offloading the aggregation of Kubernetes object states to a dedicated service, Metricbeat avoids putting unnecessary load on the Kubernetes API server. The metrics collected—ranging from pod states to persistent volume claims—provide a holistic view of the cluster's operational health. When this data is correlated with the host-level system metrics (CPU, Memory, Disk), administrators can perform root-cause analysis more effectively. For example, a spike in state_pod restarts can be correlated with a spike in node memory usage, indicating an OOM (Out of Memory) condition across the node.

The use of ConfigMaps for configuration management allows for seamless updates across the entire DaemonSet. When a configuration change is required, updating the ConfigMap and triggering a rollout of the DaemonSet ensures that all agents are synchronized. This is far more efficient than manual updates on individual nodes.

The overall synergy between Metricbeat, Elasticsearch, and Kibana creates a closed-loop monitoring system. The "self-monitoring" aspect—where the tools used for monitoring are themselves monitored by the same architecture—ensures that the monitoring pipeline is stable. If the Elasticsearch cluster's performance degrades, the Metricbeat agents will capture this in the node statistics and shard allocation metrics, which will then be visualized in Kibana, alerting the operator to the issue before it results in a catastrophic failure.

Sources

1. OneUptime
2. Vocon IT
3. Elastic Guide
4. Elastic Examples GitHub
5. Elastic Beats GitHub