The deployment of an Elastic Stack (ELK) architecture within Google Cloud Platform (GCP) represents a sophisticated approach to observability and security information and event management (SIEM). By integrating Elasticsearch, Logstash, and Kibana, organizations can transform raw telemetry from Google Cloud resources into actionable intelligence. This integration allows for the centralized monitoring of diverse GCP services, including Compute Engine, App Engine, Dataflow, Dataproc, and BigQuery. The fundamental objective is to bridge the gap between Google Operations suite—formerly known as Stackdriver—and the Elastic ecosystem, creating a unified observability plane that spans from cloud-native environments to on-premises infrastructure. This deep-dive exploration details the technical requirements, the infrastructure-as-code (IaC) deployment using OpenTofu, the configuration of data pipelines via Filebeat and Logstash, and the eventual visualization of security data within Kibana.
Architectural Framework and Data Flow Orchestration
The movement of data from a Google Cloud resource to an Elastic Stack dashboard is not a direct transfer but a multi-stage pipeline designed for reliability and scalability. The architectural flow begins at the resource level, where Google Cloud generates audit logs, firewall logs, and Virtual Private Cloud (VPC) flow logs.
The high-level data flow operates as follows:
- Google Cloud Resources: Services such as Compute Engine or BigQuery generate logs.
- Google Cloud Operations Suite: This acts as the central repository receiving logs, metrics, and traces.
- Log Sinks: Specific sinks are created within Google Cloud to filter and route desired logs.
- Google Pub/Sub: The sinks push data into Pub/Sub topics, which serve as the asynchronous messaging layer.
- Data Shippers (Filebeat/Logstash): These components subscribe to the Pub/Sub topics to ingest the data.
- Elasticsearch: The ingested data is indexed and stored for high-speed search.
- Kibana: The final layer provides the visualization and analysis interface.
This structure ensures that if the Elastic Stack is temporarily unavailable, the Pub/Sub layer retains the data, preventing loss of critical security telemetry.
Infrastructure Deployment via OpenTofu
For a Proof of Concept (PoC) or production deployment, utilizing Infrastructure as Code (IaC) via OpenTofu ensures consistency and repeatability. The deployment focuses on isolating the database and processing layers while maintaining controlled access to the visualization layer.
Core Infrastructure Configuration
The main.tf file defines the provider and the network topology. A dedicated Virtual Private Cloud (VPC) is created to isolate the Elastic Stack components.
```terraform
provider "google" {
project = "your-project-id" # Replace with Google Cloud Project ID
region = "us-central1" # Replace with your chosen region
}
resource "googlecomputenetwork" "elasticstacknetwork" {
name = "elastic-stack-network"
}
```
Compute Engine Instance Specifications
The architecture utilizes three distinct virtual machines, each tailored to its specific role in the stack. All instances are deployed using the debian-cloud/debian-10 image.
| Component | Machine Type | Disk Size | Network Configuration | Purpose |
|---|---|---|---|---|
| Elasticsearch VM | e2-standard-4 | 100 GB | Private Subnet (No Public IP) | Indexing and Storage |
| Logstash VM | e2-standard-4 | 100 GB | Private Subnet (No Public IP) | Data Processing and Ingestion |
| Kibana VM | e2-standard-4 | 50 GB | Publicly Accessible (Restricted) | Visualization Interface |
The Elasticsearch and Logstash VMs are intentionally configured without public IP addresses (access_config = []) to minimize the attack surface, ensuring that these backend components are not reachable from the open internet.
```terraform
resource "googlecomputeinstance" "elasticsearchvm" {
name = "elasticsearch-vm"
machinetype = "e2-standard-4"
zone = "us-central1-a"
bootdisk {
initializeparams {
image = "debian-cloud/debian-10"
size = 100
}
}
networkinterface {
network = googlecomputenetwork.elasticstacknetwork.id
accessconfig = []
}
}
resource "googlecomputeinstance" "logstashvm" {
name = "logstash-vm"
machinetype = "e2-standard-4"
zone = "us-central1-a"
bootdisk {
initializeparams {
image = "debian-cloud/debian-10"
size = 100
}
}
networkinterface {
network = googlecomputenetwork.elasticstacknetwork.id
accessconfig = []
}
}
resource "googlecomputeinstance" "kibanavm" {
name = "kibana-vm"
machinetype = "e2-standard-4"
zone = "us-central1-a"
bootdisk {
initializeparams {
image = "debian-cloud/debian-10"
size = 50
}
}
networkinterface {
network = googlecomputenetwork.elasticstack_network.id
}
}
```
Security and Firewall Configuration
To allow administrative access to the Kibana dashboard while maintaining security, a specific firewall rule is implemented. This rule restricts traffic to TCP port 5601, only allowing requests from a specific, authorized IP range.
terraform
resource "google_compute_firewall" "allow_kibana" {
name = "allow-kibana"
network = google_compute_network.elastic_stack_network.name
allow {
protocol = "tcp"
ports = ["5601"]
}
source_ranges = ["YOUR_ALLOWED_IP_RANGE/32"]
}
The application of this configuration is performed using the following terminal commands:
bash
tofu apply
Verification of the deployed instances is handled via the Google Cloud SDK:
bash
gcloud compute instances list
Component Installation and Software Configuration
Once the infrastructure is provisioned, each VM must be configured with its respective Elastic Stack software.
Elasticsearch Installation and Tuning
Elasticsearch serves as the heart of the stack, providing the distributed search and analytics engine.
Installation process:
bash
sudo apt update && sudo apt install -y elasticsearch
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch
To ensure that Elasticsearch is reachable by Logstash and Kibana within the private network, the elasticsearch.yml configuration file must be modified:
```yaml
/etc/elasticsearch/elasticsearch.yml
network.host: 0.0.0.0
discovery.seedhosts: ["localhost"]
cluster.initialmaster_nodes: ["node-1"]
```
After applying these changes, the service must be restarted:
bash
sudo systemctl restart elasticsearch
Verification of the Elasticsearch health status is performed using a curl request to the local API:
bash
curl -X GET "http://localhost:9200/"
Logstash Integration for Firewall Logs
Logstash acts as the server-side data processing pipeline. In this SIEM context, it is configured to ingest firewall logs directly from Google Pub/Sub.
Installation process:
bash
sudo apt update && sudo apt install -y logstash
sudo systemctl enable logstash
The ingestion pipeline is defined in a configuration file, where the google_pubsub input plugin is utilized. This requires a service account JSON key for authentication.
```yaml
/etc/logstash/conf.d/firewall_logs.conf
input {
googlepubsub {
projectid => "your-project-id"
topic => "firewall-logs"
jsonkeyfile => "/path/to/yourserviceaccount.json"
}
}
output {
elasticsearch {
hosts => ["http://elasticsearch-vm-ip:9200"]
index => "firewall-logs"
}
}
```
To activate the pipeline:
bash
sudo systemctl start logstash
Verification of the operational status:
bash
sudo systemctl status logstash
Advanced Data Ingestion with Filebeat
Filebeat is a lightweight shipper designed to send data from edge nodes to Elasticsearch. In the context of Google Cloud, Filebeat utilizes a specialized module to streamline the ingestion of operational data.
Google Cloud Module Configuration
The googlecloud module in Filebeat is specifically engineered to interface with Google Cloud's logging infrastructure. To begin, the module must be enabled:
bash
filebeat modules enable googlecloud
The configuration involves mapping the Filebeat instance to the correct GCP project and Pub/Sub topics. The JSON credentials file for the service account must be placed in /etc/filebeat/. The configuration file /etc/filebeat/modules.d/googlecloud.yml is then updated to reflect the project's specific values.
Example configuration for VPC Flow and Audit logs:
yaml
- module: googlecloud
vpcflow:
enabled: true
var.project_id: els-dummy
var.topic: els-gcp-vpc-flow-logs
audit:
enabled: true
var.project_id: els-dummy
var.topic: els-gcp-audit-logs
var.subscription_name: els-gcp-audit-logs-sub
var.credentials_file: /etc/filebeat/kdr-gcp-logs-sa-editor-only.json
Execution and Monitoring
Filebeat is started using the system service manager. Using the -e flag allows the operator to view logs in the console for real-time troubleshooting of the connection between the Google Cloud subscription and the Elasticsearch cluster.
bash
sudo service filebeat start -e
Analysis and Visualization in Kibana
The culmination of the data pipeline is the visualization layer in Kibana. Once Filebeat or Logstash begins shipping data, the indices are created in Elasticsearch, allowing Kibana to generate interactive dashboards.
Utilizing Prebuilt Dashboards
The Google Cloud module for Filebeat provides pre-configured dashboards that eliminate the need for manual visualization building. These dashboards are accessible via the "Dashboard" section of the Kibana side navigation.
Key visualizations available in the Google Cloud "Audit" dashboard include:
- Dynamic maps showing the geographic source locations of events.
- Time-series analysis of event outcomes (e.g., success vs. failure).
- Breakdown of specific event actions performed within the GCP environment.
These tools allow security analysts to identify anomalies, such as unauthorized access attempts or unusual API calls, by observing spikes in the event outcome graphs or unexpected geographic origins on the dynamic map.
Comprehensive Comparison of Ingestion Methods
Depending on the use case, an organization may choose between Logstash and Filebeat for GCP ingestion.
| Feature | Logstash Method | Filebeat Method |
|---|---|---|
| Resource Overhead | High (JVM based) | Low (Go based) |
| Transformation | Advanced filtering/mutation | Simple shipping/module-based |
| Deployment | Centralized server | Distributed edge shipper |
| Configuration | Complex .conf files |
YAML based modules |
| Use Case | Heavy data transformation | High-volume log shipping |
Conclusion
The implementation of the Elastic Stack on Google Cloud transforms the raw logs of the Google Operations suite into a high-fidelity security monitoring system. By leveraging OpenTofu for infrastructure orchestration, the deployment achieves a high level of security through the use of private subnets for Elasticsearch and Logstash, while strictly controlling access to Kibana via targeted firewall rules.
The integration of Google Pub/Sub as the intermediary transport layer ensures that the system is resilient to spikes in log volume and prevents data loss during network partitions. The use of the Filebeat googlecloud module further simplifies the process, providing immediate visibility through prebuilt dashboards that visualize audit and VPC flow logs. This architecture not only centralizes Google Cloud telemetry but also allows it to be correlated with other observability data from on-premises or hybrid environments, providing a comprehensive, unified view of the entire organizational infrastructure.