The implementation of an Elastic Stack (ELK) on Google Cloud Platform (GCP) represents a sophisticated approach to centralized logging, real-time data analytics, and infrastructure monitoring. By integrating Elasticsearch, Kibana, and the Elastic Agent—orchestrated through the Elastic Cloud on Kubernetes (ECK) operator—organizations can transform raw telemetry into actionable insights. This architecture leverages the elasticity of Google Kubernetes Engine (GKE) and the streaming capabilities of Cloud Pub/Sub and Dataflow to create a high-throughput pipeline capable of ingesting millions of events per second. The synergy between these tools allows for a seamless flow from data generation in GCP services (such as VPC Flow logs or Audit logs) to visualization in Kibana, providing a comprehensive view of the cloud environment's operational health and security posture.
Orchestrating the Elastic Stack with ECK
The deployment of the Elastic Stack on Kubernetes is streamlined through the Elastic Cloud on Kubernetes (ECK) operator. The operator serves as the control plane for the Elastic Stack, automating the lifecycle management of Elasticsearch and Kibana clusters, including upgrades, scaling, and health monitoring.
The installation process begins with the deployment of the ECK Helm chart. This ensures that the operator is correctly configured with the necessary resource constraints to prevent node instability.
helm upgrade --install "elastic-operator" "elastic/eck-operator" \ --version="2.8.0" \ --create-namespace \ --namespace="elastic-system" \ --set="resources.limits.cpu=250m" \ --set="resources.limits.memory=512Mi" \ --set="resources.limits.ephemeral-storage=1Gi" \ --set="resources.requests.cpu=250m" \ --set="resources.requests.memory=512Mi" \ --set="resources.requests.ephemeral-storage=1Gi"
The technical application of these resource limits (250m CPU and 512Mi Memory) ensures that the operator does not over-consume cluster resources while maintaining enough headroom for the reconciliation loop to function. The use of ephemeral storage limits prevents the operator from filling up the node's root partition.
To verify the successful deployment of the operator, a continuous monitoring command is used:
watch kubectl get pods -n elastic-system
The expected output identifies the elastic-operator-0 pod in a Running state with 1/1 containers ready. Once this state is achieved, the operator is active and capable of interpreting Custom Resource Definitions (CRDs) to deploy the full Elastic Stack.
The Elastic Stack Component Architecture
A fully realized Elastic Stack deployment consists of three primary components: Elasticsearch, Kibana, and the Elastic Agent, often managed via Fleet.
- Elasticsearch: This is the heart of the stack, serving as a distributed, RESTful search and analytics engine. It indexes data in near real-time, allowing for complex queries across massive datasets.
- Kibana: An open-source data analytics and visualization tool. It acts as the window into Elasticsearch, providing the UI for searching, analyzing, and creating dashboards from the indexed data.
- Elastic Agent: A lightweight, unified data shipper. Unlike legacy setups that required separate installations of Filebeat, Metricbeat, and Packetbeat, the Elastic Agent collects logs and metrics from diverse sources and transmits them to Elasticsearch.
- Elastic Fleet: A management mode where agents report to a central fleet server. This allows administrators to push configuration changes to thousands of agents simultaneously without manual intervention on each endpoint.
The integration of these components creates a scalable solution. For instance, the Elastic Agent collects a log, Elasticsearch indexes it, and Kibana visualizes the trend, all while Fleet ensures the agent is running the correct version and configuration.
Network Infrastructure and DNS Configuration
To make the Elastic Stack accessible and secure via HTTPS, a robust networking layer must be established using Google Cloud's native services. This involves the allocation of a global IP and the configuration of Cloud DNS.
The process starts with the creation of a global external IP address to serve as the entry point for the stack:
gcloud compute addresses create "elastic-stack" --global
Following the IP allocation, a managed DNS zone and record set are created to map a human-readable domain to that IP address:
gcloud dns managed-zones create "elk" \ --description="DNS Zone for Airflow" \ --dns-name="elk.BASE_DOMAIN" \ --visibility="public"
gcloud dns record-sets create "elk.BASE_DOMAIN" \ --rrdatas="$(gcloud compute addresses describe "elastic-stack" --global --format="value(address)")" \ --ttl="300" \ --type="A" \ --zone="elk"
The TTL (Time to Live) of 300 seconds ensures that DNS changes propagate quickly. To complete the delegation of the DNS zone as a subdomain of the base domain, name servers are retrieved:
gcloud dns record-sets describe elk.BASE_DOMAIN \ --type="NS" \ --zone="elk" \ --format="value(DATA)"
This network setup is critical because it allows the ManagedCertificate within the Kubernetes ingress to validate the domain ownership and provision a valid SSL certificate, ensuring that all traffic between the user's browser and Kibana is encrypted via TLS.
Deploying Ingress and Elastic Agents
Once the network and operator are in place, the actual workloads are deployed using Kubernetes manifests. The ingress configuration is applied to enable external access to the Kibana dashboard:
kubectl apply -f ingress.yaml
The deployment of the Elastic Agent and the Fleet Server is handled by a specialized manifest that defines how the agents interact with the operator:
kubectl apply -f fleet-server-and-agents.yaml
The operational status of these agents is monitored using the following command:
watch kubectl --namespace elastic-system get agents.agent.k8s.elastic.co
A successful deployment is indicated by a green health status for both the elastic-agent and the fleet-server. The health status "green" implies that the agent has successfully checked into the fleet server and is ready to receive configuration policies.
Advanced Logging and Monitoring Integration
To achieve full observability, the Elastic Stack can be integrated with kube-state-metrics. This allows the stack to ingest cluster-level metrics, providing visibility into the health of the Kubernetes nodes and pods themselves.
The installation of the Prometheus community chart is required:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm install kube-state-metrics prometheus-community/kube-state-metrics --namespace elastic-system
With this integration, Elasticsearch can store metrics regarding pod CPU usage, memory pressure, and node availability, which are then visualized in Kibana. To access the Kibana UI, administrators must retrieve the default elastic user credentials from the Kubernetes secret:
kubectl get secret elasticsearch-es-elastic-user -o yaml -n elastic-system -o jsonpath='{.data.elastic}' | base64 -d
The user then navigates to https://elk.BASE_DOMAIN to begin the configuration of dashboards.
High-Throughput Data Ingestion via Dataflow and Pub/Sub
For large-scale GCP environments, relying on simple pollers is inefficient. Instead, a streaming architecture using Cloud Pub/Sub and Cloud Dataflow is recommended to move data from GCP services into Elasticsearch.
The architectural flow is as follows:
GCP Service (e.g., Audit Logs) -> Cloud Pub/Sub Topic -> Cloud Dataflow (Pub/Sub to Elasticsearch Template) -> Elasticsearch.
Configuring the GCP Integration in Kibana
Before the data can be visualized, the GCP integration must be enabled in the Kibana UI:
1. Navigate to 'Integrations' and search for 'GCP'.
2. Select 'Google Cloud Platform (GCP)' and click 'Add Google Cloud Platform (GCP)'.
3. Crucially, uncheck the 'Collect Google Cloud Platform (GCP) … (input: gcp-pubsub)' option. This is necessary because the data is being pushed via a Dataflow pipeline rather than being pulled by a poller.
Authenticating the Pipeline
The Dataflow pipeline requires secure authentication to write data into Elasticsearch. This is handled via an API key:
- Navigate to 'Stack Management' > 'API keys' in Kibana to generate a new key.
- The base64-encoded API key must be stored securely and provided to the Dataflow job.
- The Cloud ID, found under 'Cloud' > 'Deployments' in the Elastic Cloud UI, is also required to route the data to the correct cluster.
Executing the Dataflow Job
The deployment can be performed via the Google Cloud Console:
1. Navigate to the Dataflow Jobs page.
2. Click 'Create Job from Template'.
3. Select the 'Cloud Pub/Sub to Elasticsearch' template.
4. Specify the Pub/Sub topic (e.g., my-logs).
5. In the 'Optional Parameters', set 'The type of logs' to audit. This ensures the data is categorized correctly to populate the GCP audit dashboard.
Upon running the job, the pipeline streams events. In high-load scenarios, the WriteToElasticsearch step can handle significant throughput, such as 2,800 elements per second.
Data Schema and Mapping for GCP Integrations
The Elastic GCP integration utilizes a specific schema to ensure that logs from various GCP services are consistent and searchable. This is achieved through a structured set of keywords and data types.
GCP VPC Flow Log Schema
VPC Flow logs provide detailed information about the network traffic within a VPC. The data is structured as follows:
| Field | Description | Type |
|---|---|---|
gcp.source.instance.project_id |
ID of the project containing the source VM | keyword |
gcp.source.instance.region |
Region of the source VM | keyword |
gcp.source.instance.zone |
Zone of the source VM | keyword |
gcp.source.vpc.vpc_name |
The name of the VPC | keyword |
gcp.vpcflow.rtt.ms |
Round trip time in milliseconds | numeric |
input.type |
The input method (e.g., gcp-pubsub) |
keyword |
log.offset |
The offset of the log entry | long |
An example of a processed VPC Flow log document includes the @timestamp, the agent details (version 8.18.0), and the destination geographic data (e.g., country_name: Bhutan), allowing users to visualize network traffic on a global map.
GCP Audit Log Schema
Audit logs track "who did what, where, and when" within the GCP project. The mapping for these logs is highly granular:
| Field | Description | Type |
|---|---|---|
gcp.audit.service_name |
The API service performing the operation (e.g., datastore.googleapis.com) |
keyword |
gcp.audit.status.code |
The status code (enum of google.rpc.Code) |
integer |
gcp.audit.status.message |
Developer-facing error message in English | keyword |
gcp.audit.status.details |
List of messages containing error details | flattened |
gcp.audit.type |
The type of the audit event | keyword |
gcp.audit.source_log_ids.log_time |
The timestamp of the log | date |
The use of the flattened type for gcp.audit.status.details is particularly important, as it allows Elasticsearch to index a variable number of error details without causing a mapping explosion.
Analysis and Conclusion
The deployment of an ELK stack on GCP via the ECK operator represents a shift toward "Infrastructure as Code" for observability. By utilizing the ECK operator, the complexity of managing Elasticsearch shards, indices, and Kibana configurations is abstracted into Kubernetes manifests. The technical decision to use a Dataflow pipeline for ingestion instead of direct polling solves the scalability bottleneck often found in large-scale GCP environments.
The integration of the GCP-specific schema ensures that the data is not just stored, but is contextually aware of the GCP hierarchy (Project -> Region -> Zone). This allows an administrator to query all audit logs for a specific project or all VPC flow logs for a specific zone across the entire organization.
From a performance perspective, the use of resource limits in the Helm chart and the strategic use of kube-state-metrics ensures that the monitoring system itself does not become a point of failure. The resulting architecture—combining GKE, Cloud DNS, Pub/Sub, Dataflow, and the Elastic Stack—provides a production-grade observability platform capable of handling the rigorous demands of modern cloud-native applications.