The deployment of the Elastic Stack—comprising Elasticsearch, Logstash, and Kibana—has transitioned from monolithic server installations to containerized orchestration. While early iterations of containerized logging solutions favored single-container bundles for simplicity, modern infrastructure demands a more granular approach. The prevailing consensus among DevOps engineers and system architects is that running each component of the ELK stack in its own isolated container provides superior scalability, maintainability, and resource management. This separation allows each service to scale independently based on its specific workload characteristics: Elasticsearch for storage and query performance, Logstash for data ingestion rates, and Kibana for interactive user sessions.
Implementing this architecture requires careful attention to networking, configuration file management, and initialization sequences. Whether utilizing community-maintained repositories like deviantony/docker-elk or crafting custom Docker Compose configurations from scratch, the underlying principles of container isolation, bind mount integrity, and network connectivity remain critical for a stable production environment.
The Case for Multi-Container Architecture
When considering the deployment of the ELK stack on Docker, administrators are often presented with two primary architectural patterns. The first involves using a pre-bundled image, such as sebp/elk, which runs Elasticsearch, Logstash, and Kibana within a single container. While this approach reduces the initial setup complexity, it introduces significant limitations in a production environment. A single container binds the lifecycle and resource allocation of all three services together, preventing independent scaling or updates.
The recommended approach for production-level setups, particularly those handling substantial data volumes, is to run each service in its own container. This three-container model addresses the distinct scaling properties of each component. Elasticsearch nodes may need to be scaled horizontally to increase storage capacity, redundancy, and search performance. Logstash containers scale based on the rate of data ingestion, requiring more compute power when processing high-velocity logs. Kibana instances are primarily scaled to handle concurrent interactive queries from users. By decoupling these services, administrators can optimize resource usage and ensure that a heavy load on the ingestion pipeline does not directly impact the availability of the visualization interface.
For environments processing approximately 20GB of log data daily from numerous instances and applications, the multi-container approach ensures that the infrastructure can evolve to meet growing demands without requiring a complete redeployment of the entire stack.
Prerequisites and Host Preparation
Before deploying the ELK stack, the underlying host system must be prepared. A typical production deployment occurs on a server running Ubuntu 22.04 or later. The host requires SSH access and root or sudo privileges to install necessary packages. Basic familiarity with Docker, Docker Compose, Elasticsearch configurations, and YAML syntax is essential for troubleshooting and customization.
The installation process begins by updating the system packages and installing curl, which is required to fetch the Docker installation script. The command sudo apt-get update && sudo apt-get install curl -y ensures that the package index is current and that the curl utility is available. Once curl is installed, the official Docker installation script can be executed. The command curl https://get.docker.com | sh downloads the script from Docker’s official distribution channel and pipes it directly into the sh shell for execution. This script handles the configuration of repositories and the installation of Docker Engine and Docker Compose, providing a streamlined path to a functional containerization platform.
Cloning and Initializing the Stack
For those leveraging community resources, the deviantony/docker-elk repository offers a robust starting point. To deploy this stack, the repository must first be cloned onto the Docker host using the command git clone https://github.com/deviantony/docker-elk.git. It is crucial to ensure that the repository is cloned in a location supported by the Docker daemon, or to follow specific documentation if additional locations are required.
Initialization is not automatic and requires specific setup steps to establish the necessary Elasticsearch users and groups. Executing docker compose up setup triggers the initialization process. This step is critical for security, as it configures the built-in users required for authentication.
Security best practices dictate the generation of encryption keys for Kibana. This is achieved by running docker compose up kibana-genkeys. The output of this command must be copied into the Kibana configuration file, specifically kibana/config/kibana.yml. This step ensures that data transmitted between Kibana and Elasticsearch is encrypted, protecting sensitive log data and authentication credentials.
Once the setup and key generation are complete, the remaining stack components can be started with docker compose up. For production environments where the terminal is not intended to remain attached to the process, the -d flag should be appended to run the services in detached mode: docker compose up -d.
Configuration Management and Bind Mounts
When constructing a custom ELK stack or modifying existing configurations, precise management of configuration files is paramount. Docker bind mounts expect the host files to already exist. If a file specified in the docker-compose.yml does not exist on the host, Docker will automatically create a directory in its place. This behavior often leads to mount errors when the container attempts to write to a file path that the host has interpreted as a directory.
To prevent this, administrators must manually create the necessary directory structures and ensure that configuration files are actual files, not directories. The following directory structure is typical for a modular ELK setup:
elk-stack/logstash/pipelineelk-stack/logstash/configelk-stack/elasticsearchelk-stack/kibana
Within this structure, individual configuration files must be created using a text editor such as nano. The critical files include docker-compose.yml, elasticsearch/elasticsearch.yml, logstash/config/logstash.yml, logstash/pipeline/logstash.conf, and kibana/kibana.yml.
After creation, it is vital to verify that these are regular files. Running ls -la logstash/config/logstash.yml should return permissions starting with - (e.g., -rw-r--r--), indicating a file. If the output starts with d (e.g., drwxr-xr-x), the path is a directory, which will cause the container to fail. This verification step is a common troubleshooting checkpoint when containers refuse to start due to configuration volume errors.
Networking and Service Discovery
Effective communication between the ELK components and external log shippers relies on proper Docker networking. Legacy approaches using container links are deprecated and should be avoided in favor of user-defined networks. A modern setup involves creating a network, such as elknet, and attaching all relevant containers to it.
When starting the ELK container, the command might resemble:
bash
sudo docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk --network=elknet sebp/elk
In this scenario, the container named elk is connected to the elknet network. Any other container, such as an application server running Filebeat, that is also connected to elknet can resolve the hostname elk to the IP address of the ELK container. This simplifies the configuration of log shippers, as they only need to specify elk as the host in their filebeat.yml configuration.
For more complex setups involving multiple Elasticsearch nodes, specific configurations are required to enable cluster discovery. In elasticsearch.yml, settings such as network.host: 0.0.0.0 and discovery.zen.ping.unicast.hosts: ["elk"] ensure that nodes can find and communicate with each other. When running additional Elasticsearch nodes, care must be taken not to duplicate port mappings on the host. For instance, if the primary ELK container publishes port 9200, a secondary node should not publish the same port to the host unless a proxy or load balancer is in place. Instead, the internal Docker network handles the communication between nodes.
Operational Lifecycle and Maintenance
Once the stack is deployed, operational commands are necessary for maintenance and updates. A critical rule when working with versioned stacks like docker-elk is that images must be rebuilt whenever switching branches or updating the version of an existing stack. Failure to run docker compose build after such changes can result in outdated configurations or runtime errors.
Accessing the Kibana web UI typically occurs at http://localhost:5601. Upon initial startup, the system initializes default users. The default credentials are elastic for the username and changeme for the password. These values are derived from the .env file. Administrators should replace these default credentials immediately to secure the instance.
Monitoring the health of the stack involves checking the logs and cluster status. The command docker compose logs -f elasticsearch allows administrators to watch the startup process, which typically takes 30 to 60 seconds for Elasticsearch to become fully healthy. Verification can be performed using curl http://localhost:9200/_cluster/health?pretty, which provides a detailed view of the cluster’s status, node count, and shard allocation.
When decommissioning the stack, the command docker compose down stops and removes the containers and the associated network. The output confirms the removal of each container, such as logstash, elasticsearch, kibana, setup, and the network elk-stack_default. This clean teardown is essential for resetting the environment or upgrading to a new version without residual configuration conflicts.
Conclusion
The deployment of the ELK stack via Docker Compose represents a significant shift from traditional monolithic installations to a modular, scalable architecture. By separating Elasticsearch, Logstash, and Kibana into distinct containers, organizations can align their infrastructure with the specific scaling requirements of each service. This approach mitigates the risks associated with single-point-of-failure containers and allows for more granular control over resources and security.
Key to this success is the rigorous management of configuration files, ensuring that bind mounts reference actual files rather than accidentally created directories. Furthermore, leveraging user-defined networks simplifies service discovery and enables seamless integration with log shippers like Filebeat. While initial setup requires attention to detail—such as initializing users, generating encryption keys, and verifying network connectivity—the resulting infrastructure is robust, maintainable, and well-suited for handling the demands of modern application logging. As data volumes grow and security requirements tighten, this decoupled architecture provides the flexibility needed to adapt without disrupting operations.