The Elastic Stack, commonly referred to as the ELK stack, represents a sophisticated ecosystem designed for the ingestion, storage, and visualization of massive datasets, primarily focused on centralized logging. At its core, the stack is comprised of three primary pillars: Elasticsearch, Logstash, and Kibana. Elasticsearch serves as the heart of the system, acting as a distributed search and index engine that stores logs and allows for near real-time retrieval. Logstash functions as the data processing pipeline, capable of ingesting data from diverse sources, transforming it via filters, and shipping it to the indexer. Kibana provides the operational window, offering a powerful web-based interface for exploring the indexed data and constructing complex analytical dashboards.
Deploying these components via Docker Compose transforms a complex distributed system into a portable, reproducible environment. By utilizing containerization, developers and architects can prototype log pipelines, test complex Grok patterns in Logstash, and design Kibana dashboards without the overhead of provisioning dedicated physical or virtual infrastructure. This approach significantly lowers the barrier to entry for proof-of-concept (POC) projects and local development, allowing for rapid iteration of logging strategies before they are promoted to production environments.
Architectural Framework and Data Flow
The ELK stack operates on a linear data progression model where information moves from the source to the end-user interface through a series of specialized stages.
The typical flow of data begins with the Application Logs. These logs are generated by the software services and are either sent directly to Logstash or collected by an intermediary agent known as Filebeat. Filebeat is an optional but highly recommended lightweight shipper that resides on the application host, monitoring log files and forwarding them to the central Logstash instance.
Once the data reaches Logstash, it enters the transformation phase. Logstash processes the raw strings of log data, applying filters to parse the content into structured fields (such as timestamps, severity levels, and error codes). This transformation is critical because it turns unstructured text into a searchable database format. Following transformation, Logstash ships the structured data to Elasticsearch.
Elasticsearch then indexes the data, creating an inverted index that allows for lightning-fast searches across millions of records. Finally, Kibana queries the Elasticsearch API to retrieve this data, presenting it to the user through a graphical user interface.
The architectural relationship can be visualized as follows:
- Application Logs -> Logstash
- Filebeat (Optional) -> Logstash
- Logstash -> Elasticsearch
- Elasticsearch -> Kibana
Implementation Strategies for Docker Compose
There are two primary paths for deploying the ELK stack using Docker: a manual configuration approach and the use of pre-configured templates like the docker-elk repository.
Manual Configuration and Directory Setup
For those requiring a bespoke setup, a manual directory structure is necessary to handle Docker bind mounts. A critical technical detail is that Docker bind mounts expect the host files to exist prior to container startup. If a file is missing, Docker will erroneously create a directory in its place, which leads to mount errors and container crashes.
To prevent this, the following directory structure must be initialized on the host machine:
bash
mkdir -p elk-stack/logstash/pipeline
mkdir -p elk-stack/logstash/config
mkdir -p elk-stack/elasticsearch
mkdir -p elk-stack/kibana
cd elk-stack
Once the structure is created, the specific configuration files must be populated. These include the docker-compose.yml file, the elasticsearch/elasticsearch.yml configuration, the logstash/config/logstash.yml for general settings, the logstash/pipeline/logstash.conf for data processing logic, and the kibana/kibana.yml for the visualization settings. It is imperative to verify that these are regular files and not directories before starting the stack. This can be verified using the following command:
bash
ls -la logstash/config/logstash.yml
The expected output should show a file permission string such as -rw-r--r--, indicating a file. If the output shows drwxr-xr-x, it is a directory and must be deleted and recreated as a file.
Utilizing the docker-elk Template
For users seeking a rapid deployment, the docker-elk repository by deviantony provides a streamlined template. This project is designed as a tool for tweaking and exploration rather than a production blueprint. It prioritizes minimal, unopinionated configurations and high-quality documentation over complex automation.
To implement this stack, the repository must be cloned onto the Docker host:
bash
git clone https://github.com/deviantony/docker-elk.git
The initialization process requires a specific sequence of commands to ensure the internal security and user accounts are properly set up. First, the Elasticsearch users and groups must be initialized:
bash
docker compose up setup
For enhanced security, it is highly recommended to generate encryption keys for Kibana:
bash
docker compose up kibana-genkeys
The output from the key generation command must be manually copied into the kibana/config/kibana.yml file. Once these steps are completed, the entire stack can be launched:
bash
docker compose up
To run the services in the background (detached mode), use the following command:
bash
docker compose up -d
Network Configuration and Port Mapping
The ELK stack relies on several specific ports for communication between components and for external access. Understanding these mappings is essential for firewall configuration and troubleshooting.
| Component | Port | Purpose |
|---|---|---|
| Logstash | 5044 | Beats input for Filebeat/Metricbeat |
| Logstash | 50000 | General TCP input |
| Logstash | 9600 | Monitoring API |
| Elasticsearch | 9200 | HTTP API (REST) |
| Elasticsearch | 9300 | TCP Transport (Node-to-Node) |
| Kibana | 5601 | Web User Interface |
Security, Authentication, and Access
Upon the initial startup of the stack, the system initializes three primary Elasticsearch users: elastic, logstash_internal, and kibana_system. By default, these users are assigned the password changeme, as defined in the .env file.
To access the Kibana web interface, the user should navigate to http://localhost:5601 and authenticate using the following credentials:
- User: elastic
- Password: changeme
For those requiring higher security, the tls variant of the stack is available, which enables TLS encryption across Elasticsearch, Kibana, and Fleet.
Regarding licensing, the stack enables Platinum features by default for a 30-day trial period. After this trial expires, the system seamlessly transitions to the Open Basic license. This transition occurs without manual intervention and ensures that no data is lost during the downgrade to the free feature set.
System Optimization and Resource Management
The ELK stack is notoriously memory-intensive, as Elasticsearch and Logstash both run on the Java Virtual Machine (JVM). Failure to tune the JVM heap sizes often leads to "Out of Memory" (OOM) kills by the Docker daemon or the host operating system.
Memory Tuning for Development
For local development on hardware with limited resources, the JVM heap sizes should be reduced in the docker-compose.yml file:
yaml
elasticsearch:
environment:
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
logstash:
environment:
- "LS_JAVA_OPTS=-Xms256m -Xmx256m"
Memory Requirements for Production
In a production environment, the resource requirements increase significantly to maintain stability and performance. Elasticsearch should be allocated a minimum of 4GB of heap, and Logstash should be allocated at least 1GB.
Index Lifecycle Management (ILM)
To prevent Elasticsearch from consuming all available disk space in long-running deployments, Index Lifecycle Management must be configured. This involves creating a policy that automatically deletes logs after a certain period. For example, a policy that deletes logs older than 30 days can be created using the following curl command:
bash
curl -X PUT "http://localhost:9200/_ilm/policy/logs-policy" -H "Content-Type: application/json" -d '{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"rollover": {
"max_size": "5gb",
"max_age": "1d"
}
}
},
"delete": {
"min_age": "30d",
"actions": {
"delete": {}
}
}
}
}
}'
Troubleshooting and Operational Health
Monitoring the health of the ELK stack is critical for ensuring data integrity and availability. Because the containers start asynchronously, it is important to monitor the logs to confirm that Elasticsearch is healthy before attempting to use Kibana.
The logs can be followed using:
bash
docker compose logs -f elasticsearch
Once the service is running, several API endpoints can be used to verify the cluster status:
Cluster Health Check:
bash curl http://localhost:9200/_cluster/health?prettyIndex Listing:
bash curl http://localhost:9200/_cat/indices?vLogstash Pipeline Statistics:
bash curl http://localhost:9600/_node/stats/pipelines?prettyIdentifying Processing Errors:
To find specific errors within the Logstash logs, use the following command:
bash
docker compose logs logstash | grep -i error
Host-Specific Configuration Warnings
The deployment of the ELK stack is subject to various host-level restrictions and requirements depending on the operating system and Docker version.
On Linux systems, it is mandatory that the user executing the Docker commands has the appropriate permissions to interact with the Docker daemon (typically by being part of the docker group). Additionally, users should be aware that in development environments, Elasticsearch bootstrap checks are often disabled to simplify setup. However, for production, the host must be configured according to the official Elasticsearch "Important System Configuration" guidelines to ensure stability.
For Windows users utilizing the legacy Hyper-V mode of Docker Desktop, File Sharing must be explicitly enabled for the C: drive to allow the containers to mount configuration files.
For Mac users, the default configuration of Docker Desktop only allows mounting files from specific directories: /Users/, /Volume/, /private/, /tmp, and /var/folders. Any attempt to mount configurations outside these paths will fail.
Conclusion
The deployment of an ELK stack via Docker Compose represents a powerful intersection of centralized logging and container orchestration. By decoupling the ingestion (Logstash), storage (Elasticsearch), and visualization (Kibana) layers into discrete containers, users gain an unprecedented level of flexibility in how they manage their telemetry data. The transition from a manual setup, which requires meticulous attention to directory structures and bind mounts, to a template-based approach like docker-elk demonstrates the evolution of the stack toward accessibility.
However, the operational success of the stack depends heavily on JVM tuning and Index Lifecycle Management. Without strict control over heap sizes and automated index deletion, the stack can quickly destabilize a host system. The ability to scale from a 512MB development footprint to a production-grade deployment with 4GB+ of heap per node allows the ELK stack to grow alongside the application it monitors. Ultimately, the use of Docker Compose for this stack serves as an ideal bridge between a local proof-of-concept and a full-scale production logging architecture, providing a safe environment for experimentation with log pipelines and data visualization.