The ELK stack, comprising Elasticsearch, Logstash, and Kibana, serves as a foundational architecture for modern log aggregation, data analysis, and visualization. Deploying this stack traditionally involves complex configuration of individual services, but leveraging Docker and Docker Compose significantly streamlines the process. This approach allows administrators to spin up a fully functional, interconnected environment with a single command, making it ideal for development, testing, and lightweight production scenarios. This guide details the precise steps to install and configure the ELK stack on an Ubuntu server using Docker Compose, drawing from established community tutorials and the official deviantony/docker-elk repository.
The architecture relies on three core components, each serving a distinct purpose in the data pipeline. Elasticsearch acts as the central search and analytics engine, indexing and storing data for high-performance retrieval. Kibana provides the user interface layer, offering visualization capabilities that allow users to analyze the data stored in Elasticsearch. Logstash serves as the data processing pipeline, capable of analyzing logs from various applications, transforming them, and forwarding them to Elasticsearch. By containerizing these services, administrators gain the ability to analyze any dataset by utilizing the searching and aggregation capabilities of Elasticsearch combined with the visualization power of Kibana.
Prerequisites and System Preparation
Before initiating the deployment, specific system prerequisites must be met to ensure compatibility and stability. The target server should be running Ubuntu version 22.04 or later. Access to the server requires SSH connectivity, along with privileges granted to the root user or a user with sudo permissions. A foundational understanding of Docker, Docker Compose, Elasticsearch, and YAML configuration syntax is assumed.
The initial step involves connecting to the server via SSH. Replace the placeholder username holu with your actual username and <your_host> with the IP address of your server.
bash
ssh holu@<your_host>
Once logged in, the system packages must be updated to ensure the latest security patches and dependencies are available. Additionally, curl is required to download the Docker installation script. Execute the following command to update the package list and install curl.
bash
sudo apt-get update && sudo apt-get install curl -y
Installing Docker and Docker Compose
With curl installed, the next phase is the installation of Docker and Docker Compose. Rather than manually configuring repositories and keys, Docker provides a quick install script that handles the entire process. This script is downloaded from get.docker.com and piped directly to the sh shell for execution. This method ensures that the latest stable versions of Docker Engine and Docker Compose are installed on the system.
bash
curl https://get.docker.com | sh
This command downloads the script and feeds it to the shell, which executes the installation routines. After the installation completes, Docker and Docker Compose are available for use. To avoid needing sudo privileges for every Docker command, the current user should be added to the docker group. Replace holu with your specific username.
bash
sudo usermod -aG docker holu
It is critical to log out and log back in for these group changes to take effect. Without this step, the user will continue to require sudo to execute Docker commands, which can complicate the workflow and pose security risks if not managed correctly.
Directory Structure and Permissions
Docker containers often store data outside the container's ephemeral filesystem to ensure data persistence across restarts. While user directories are a common location for such data, best practices often dictate storing containers in /opt/containers. This location is standard for third-party software and helps maintain a clean system structure.
Create the directory structure and set the appropriate ownership for the current user. Replace holu with your username.
bash
sudo mkdir -p /opt/containers && cd /opt/containers
sudo chown -R holu /opt/containers
Within this directory, a specific folder for the ELK stack must be created. This folder will house the docker-compose.yaml file and the esdata directory, which is designated for persistent Elasticsearch data.
bash
mkdir elk-stack && cd elk-stack && touch docker-compose.yaml && mkdir esdata && sudo chown -R 1000:1000 esdata
The esdata directory is critical for data integrity. Elasticsearch requires specific user permissions to write to its data directory. The UID 1000 is commonly used for non-root users in Docker containers. Setting the ownership of the esdata directory to 1000:1000 ensures that the Elasticsearch container can write data without permission errors. This external volume mount ensures that log data and indices survive container recreation.
Configuring Docker Compose
The docker-compose.yaml file is the central configuration hub that declares the infrastructure for the ELK stack. It defines the services, their images, environment variables, and network configurations. For a robust setup that handles security initialization, a multi-service approach is required. This includes a setup service, Elasticsearch, and Kibana. Logstash is often added depending on the specific data ingestion needs.
Below is a representative configuration snippet focusing on the setup and Elasticsearch services. This configuration uses Elasticsearch version 8.15.1, which includes significant security enhancements over previous versions.
yaml
version: "3"
services:
setup:
image: docker.elastic.co/elasticsearch/elasticsearch:8.15.1
environment:
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
- KIBANA_PASSWORD=${KIBANA_PASSWORD}
container_name: setup
command:
- bash
- -c
- |
echo "Waiting for Elasticsearch availability";
until curl -s http://elasticsearch:9200 | grep -q "missing authentication credentials"; do sleep 30; done;
echo "Setting kibana_system password";
until curl -s -X POST -u "elastic:${ELASTIC_PASSWORD}" -H "Content-Type: application/json" http://elasticsearch:9200/_security/user/kibana_system/_password -d "{\"password\":\"${KIBANA_PASSWORD}\"}" | grep -q "^{}"; do sleep 10; done;
echo "All done!";
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:8.15.1
# Configuration for elasticsearch continues here
The setup service is a temporary container that waits for Elasticsearch to become available and then configures the kibana_system user password. This automation is crucial because Elasticsearch 8.x enforces security by default, requiring passwords to be set for built-in users before Kibana can connect. The script uses curl to poll the Elasticsearch endpoint until it responds, ensuring that the password setting commands are executed only after the service is ready.
Alternative: Using the Deviantony Docker-ELK Repository
For users seeking a more feature-rich and pre-configured experience, the deviantony/docker-elk repository offers a comprehensive solution. This project is designed to make the Elastic stack as easy as possible to get into, serving as a template for exploration rather than a strict blueprint for production. The authors advocate for good documentation over elaborate automation, keeping the default configuration minimal and unopinionated.
To utilize this repository, clone it onto the Docker host.
bash
git clone https://github.com/deviantony/docker-elk.git
The repository includes various stack variants. The standard variant is used for basic deployments, while the tls variant enables TLS encryption in Elasticsearch, Kibana, and Fleet. It is important to note that Platinum features are enabled by default for a 30-day trial. After this evaluation period, the stack seamlessly reverts to the free features included in the Open Basic license without data loss or manual intervention. Users who wish to opt out of this behavior can refer to the documentation on disabling paid features.
Before starting the stack, initialize the required Elasticsearch users and groups.
bash
docker compose up setup
For enhanced security, generating encryption keys for Kibana is highly recommended. Execute the following command and copy the output to the Kibana configuration file (kibana/config/kibana.yml).
bash
docker compose up kibana-genkeys
Once the setup is complete and encryption keys are configured, start the remaining services.
bash
docker compose up
To run the services in the background, append the -d flag to the command. Upon the initial startup, the elastic, logstash_internal, and kibana_system users are initialized with passwords defined in the .env file. By default, the password for the elastic user is changeme.
Accessing Kibana and Managing the Stack
After the containers are running, allow Kibana approximately one minute to initialize its internal caches and connect to Elasticsearch. Access the Kibana web interface by opening a web browser and navigating to http://localhost:5601. Use the default credentials to log in:
- Username:
elastic - Password:
changeme
If using the deviantony repository and custom .env values were not changed, these credentials will grant access to the dashboard. From here, users can begin exploring data visualizations and setting up dashboards. If Logstash is part of the stack, users will need to create a Logstash data view in Kibana to access logs processed by Logstash.
Resource Management and Maintenance
Effective management of the ELK stack requires attention to resource allocation and lifecycle operations. Docker containers can consume significant amounts of memory, particularly Elasticsearch. To accommodate environments with limited resources, such as Docker Desktop for Mac which defaults to 2 GB of memory, the heap size allocation is capped in the docker-compose.yml file. By default, Elasticsearch is limited to 512 MB and Logstash to 256 MB.
Administrators can adjust these limits using environment variables. The startup scripts for Elasticsearch and Logstash allow the appending of extra JVM options from specific environment variables, enabling fine-tuned control over memory usage.
- Elasticsearch uses the
ES_JAVA_OPTSenvironment variable. - Logstash uses the
LS_JAVA_OPTSenvironment variable.
When switching branches or updating the version of an existing stack, it is mandatory to rebuild the images to ensure consistency.
bash
docker compose build
This command ensures that any changes in configuration files or version updates are reflected in the running containers.
To stop the stack and remove the containers, execute the following command.
bash
docker compose down
This command performs a graceful shutdown, removing the containers and the associated network. The output will confirm the removal of each service:
text
[+] Running 5/5
✔ Container logstash Removed
✔ Container elasticsearch Removed
✔ Container kibana Removed
✔ Container setup Removed
✔ Network elk-stack_default Removed
Conclusion
Deploying the ELK stack with Docker Compose offers a flexible and efficient method for implementing log aggregation and data visualization. Whether using a custom docker-compose.yaml file for a minimal setup or leveraging the comprehensive deviantony/docker-elk repository, the core principles remain the same: ensure proper system preparation, manage data persistence through external volumes, and configure security settings correctly. The transition to Elasticsearch 8.x introduces stricter security defaults, necessitating the use of setup scripts to initialize user passwords. By understanding the roles of each component and the mechanics of Docker networking and resource management, administrators can effectively deploy, maintain, and scale their ELK stack to meet diverse analytical needs. Future enhancements may include integrating log exporters like Filebeat for more robust data ingestion or exploring the TLS variant for encrypted communications.