Orchestrating Centralized Logging with the ELK Stack and Docker Compose

The Elastic Stack, commonly referred to as the ELK stack, represents a sophisticated ecosystem designed for the ingestion, storage, and visualization of massive datasets, primarily focused on centralized logging. At its core, the stack is comprised of three primary pillars: Elasticsearch, Logstash, and Kibana. Elasticsearch serves as the heart of the system, acting as a distributed search and index engine that stores logs and allows for near real-time retrieval. Logstash functions as the data processing pipeline, capable of ingesting data from diverse sources, transforming it via filters, and shipping it to the indexer. Kibana provides the operational window, offering a powerful web-based interface for exploring the indexed data and constructing complex analytical dashboards.

Deploying these components via Docker Compose transforms a complex distributed system into a portable, reproducible environment. By utilizing containerization, developers and architects can prototype log pipelines, test complex Grok patterns in Logstash, and design Kibana dashboards without the overhead of provisioning dedicated physical or virtual infrastructure. This approach significantly lowers the barrier to entry for proof-of-concept (POC) projects and local development, allowing for rapid iteration of logging strategies before they are promoted to production environments.

Architectural Framework and Data Flow

The ELK stack operates on a linear data progression model where information moves from the source to the end-user interface through a series of specialized stages.

The typical flow of data begins with the Application Logs. These logs are generated by the software services and are either sent directly to Logstash or collected by an intermediary agent known as Filebeat. Filebeat is an optional but highly recommended lightweight shipper that resides on the application host, monitoring log files and forwarding them to the central Logstash instance.

Once the data reaches Logstash, it enters the transformation phase. Logstash processes the raw strings of log data, applying filters to parse the content into structured fields (such as timestamps, severity levels, and error codes). This transformation is critical because it turns unstructured text into a searchable database format. Following transformation, Logstash ships the structured data to Elasticsearch.

Elasticsearch then indexes the data, creating an inverted index that allows for lightning-fast searches across millions of records. Finally, Kibana queries the Elasticsearch API to retrieve this data, presenting it to the user through a graphical user interface.

The architectural relationship can be visualized as follows:

  • Application Logs -> Logstash
  • Filebeat (Optional) -> Logstash
  • Logstash -> Elasticsearch
  • Elasticsearch -> Kibana

Implementation Strategies for Docker Compose

There are two primary paths for deploying the ELK stack using Docker: a manual configuration approach and the use of pre-configured templates like the docker-elk repository.

Manual Configuration and Directory Setup

For those requiring a bespoke setup, a manual directory structure is necessary to handle Docker bind mounts. A critical technical detail is that Docker bind mounts expect the host files to exist prior to container startup. If a file is missing, Docker will erroneously create a directory in its place, which leads to mount errors and container crashes.

To prevent this, the following directory structure must be initialized on the host machine:

bash mkdir -p elk-stack/logstash/pipeline mkdir -p elk-stack/logstash/config mkdir -p elk-stack/elasticsearch mkdir -p elk-stack/kibana cd elk-stack

Once the structure is created, the specific configuration files must be populated. These include the docker-compose.yml file, the elasticsearch/elasticsearch.yml configuration, the logstash/config/logstash.yml for general settings, the logstash/pipeline/logstash.conf for data processing logic, and the kibana/kibana.yml for the visualization settings. It is imperative to verify that these are regular files and not directories before starting the stack. This can be verified using the following command:

bash ls -la logstash/config/logstash.yml

The expected output should show a file permission string such as -rw-r--r--, indicating a file. If the output shows drwxr-xr-x, it is a directory and must be deleted and recreated as a file.

Utilizing the docker-elk Template

For users seeking a rapid deployment, the docker-elk repository by deviantony provides a streamlined template. This project is designed as a tool for tweaking and exploration rather than a production blueprint. It prioritizes minimal, unopinionated configurations and high-quality documentation over complex automation.

To implement this stack, the repository must be cloned onto the Docker host:

bash git clone https://github.com/deviantony/docker-elk.git

The initialization process requires a specific sequence of commands to ensure the internal security and user accounts are properly set up. First, the Elasticsearch users and groups must be initialized:

bash docker compose up setup

For enhanced security, it is highly recommended to generate encryption keys for Kibana:

bash docker compose up kibana-genkeys

The output from the key generation command must be manually copied into the kibana/config/kibana.yml file. Once these steps are completed, the entire stack can be launched:

bash docker compose up

To run the services in the background (detached mode), use the following command:

bash docker compose up -d

Network Configuration and Port Mapping

The ELK stack relies on several specific ports for communication between components and for external access. Understanding these mappings is essential for firewall configuration and troubleshooting.

Component Port Purpose
Logstash 5044 Beats input for Filebeat/Metricbeat
Logstash 50000 General TCP input
Logstash 9600 Monitoring API
Elasticsearch 9200 HTTP API (REST)
Elasticsearch 9300 TCP Transport (Node-to-Node)
Kibana 5601 Web User Interface

Security, Authentication, and Access

Upon the initial startup of the stack, the system initializes three primary Elasticsearch users: elastic, logstash_internal, and kibana_system. By default, these users are assigned the password changeme, as defined in the .env file.

To access the Kibana web interface, the user should navigate to http://localhost:5601 and authenticate using the following credentials:

  • User: elastic
  • Password: changeme

For those requiring higher security, the tls variant of the stack is available, which enables TLS encryption across Elasticsearch, Kibana, and Fleet.

Regarding licensing, the stack enables Platinum features by default for a 30-day trial period. After this trial expires, the system seamlessly transitions to the Open Basic license. This transition occurs without manual intervention and ensures that no data is lost during the downgrade to the free feature set.

System Optimization and Resource Management

The ELK stack is notoriously memory-intensive, as Elasticsearch and Logstash both run on the Java Virtual Machine (JVM). Failure to tune the JVM heap sizes often leads to "Out of Memory" (OOM) kills by the Docker daemon or the host operating system.

Memory Tuning for Development

For local development on hardware with limited resources, the JVM heap sizes should be reduced in the docker-compose.yml file:

yaml elasticsearch: environment: - "ES_JAVA_OPTS=-Xms512m -Xmx512m" logstash: environment: - "LS_JAVA_OPTS=-Xms256m -Xmx256m"

Memory Requirements for Production

In a production environment, the resource requirements increase significantly to maintain stability and performance. Elasticsearch should be allocated a minimum of 4GB of heap, and Logstash should be allocated at least 1GB.

Index Lifecycle Management (ILM)

To prevent Elasticsearch from consuming all available disk space in long-running deployments, Index Lifecycle Management must be configured. This involves creating a policy that automatically deletes logs after a certain period. For example, a policy that deletes logs older than 30 days can be created using the following curl command:

bash curl -X PUT "http://localhost:9200/_ilm/policy/logs-policy" -H "Content-Type: application/json" -d '{ "policy": { "phases": { "hot": { "min_age": "0ms", "actions": { "rollover": { "max_size": "5gb", "max_age": "1d" } } }, "delete": { "min_age": "30d", "actions": { "delete": {} } } } } }'

Troubleshooting and Operational Health

Monitoring the health of the ELK stack is critical for ensuring data integrity and availability. Because the containers start asynchronously, it is important to monitor the logs to confirm that Elasticsearch is healthy before attempting to use Kibana.

The logs can be followed using:

bash docker compose logs -f elasticsearch

Once the service is running, several API endpoints can be used to verify the cluster status:

  1. Cluster Health Check:
    bash curl http://localhost:9200/_cluster/health?pretty

  2. Index Listing:
    bash curl http://localhost:9200/_cat/indices?v

  3. Logstash Pipeline Statistics:
    bash curl http://localhost:9600/_node/stats/pipelines?pretty

  4. Identifying Processing Errors:
    To find specific errors within the Logstash logs, use the following command:

bash docker compose logs logstash | grep -i error

Host-Specific Configuration Warnings

The deployment of the ELK stack is subject to various host-level restrictions and requirements depending on the operating system and Docker version.

On Linux systems, it is mandatory that the user executing the Docker commands has the appropriate permissions to interact with the Docker daemon (typically by being part of the docker group). Additionally, users should be aware that in development environments, Elasticsearch bootstrap checks are often disabled to simplify setup. However, for production, the host must be configured according to the official Elasticsearch "Important System Configuration" guidelines to ensure stability.

For Windows users utilizing the legacy Hyper-V mode of Docker Desktop, File Sharing must be explicitly enabled for the C: drive to allow the containers to mount configuration files.

For Mac users, the default configuration of Docker Desktop only allows mounting files from specific directories: /Users/, /Volume/, /private/, /tmp, and /var/folders. Any attempt to mount configurations outside these paths will fail.

Conclusion

The deployment of an ELK stack via Docker Compose represents a powerful intersection of centralized logging and container orchestration. By decoupling the ingestion (Logstash), storage (Elasticsearch), and visualization (Kibana) layers into discrete containers, users gain an unprecedented level of flexibility in how they manage their telemetry data. The transition from a manual setup, which requires meticulous attention to directory structures and bind mounts, to a template-based approach like docker-elk demonstrates the evolution of the stack toward accessibility.

However, the operational success of the stack depends heavily on JVM tuning and Index Lifecycle Management. Without strict control over heap sizes and automated index deletion, the stack can quickly destabilize a host system. The ability to scale from a 512MB development footprint to a production-grade deployment with 4GB+ of heap per node allows the ELK stack to grow alongside the application it monitors. Ultimately, the use of Docker Compose for this stack serves as an ideal bridge between a local proof-of-concept and a full-scale production logging architecture, providing a safe environment for experimentation with log pipelines and data visualization.

Sources

  1. OneUptime Blog
  2. deviantony/docker-elk GitHub
  3. Elastic Blog

Related Posts