The implementation of a centralized logging architecture is a critical requirement for modern infrastructure management, particularly when navigating the complexities of distributed systems and microservices. The Elastic Stack, historically and commonly referred to as the ELK Stack, provides a sophisticated suite of open-source tools designed to ingest, search, analyze, and visualize log data from any source and in any format. By consolidating logs into a single, searchable interface, administrators can eliminate the inefficiency of manually logging into individual servers to inspect flat files, thereby drastically reducing the Mean Time to Resolution (MTTR) during critical system failures.
At its core, the Elastic Stack facilitates the practice of centralized logging. This process involves the aggregation of telemetry data from multiple disparate sources—ranging from kernel logs and application stdout to specialized audit trails—and forwarding them to a central repository where they are indexed for near-instantaneous retrieval. This capability is indispensable when identifying issues that span multiple servers; by correlating logs across a specific time frame, an engineer can trace a request as it traverses various network hops and services, providing a holistic view of the system's operational health.
The architecture is composed of four primary components: Elasticsearch, Logstash, Kibana, and Beats. Elasticsearch serves as the heart of the stack, acting as a highly scalable search and analytics engine. Logstash functions as the server-side data processing pipeline, capable of transforming and enriching data. Kibana provides the visualization layer, turning complex data indices into intuitive dashboards. Finally, Beats represents a family of lightweight shippers—such as Filebeat for log files and Metricbeat for system metrics—that send data from the edge to the central stack.
Technical Hardware and Software Specifications
Deploying the Elastic Stack requires a careful balance of resources, as the Java Virtual Machine (JVM) and the indexing processes of Elasticsearch are resource-intensive. Failure to provide adequate hardware will result in Out-of-Memory (OOM) errors and catastrophic performance degradation.
The following table outlines the minimum and recommended specifications for a functional deployment on Ubuntu.
| Component | Minimum RAM | Recommended RAM | CPU Requirements | Disk Space |
|---|---|---|---|---|
| Elasticsearch | 2GB+ | 4GB+ | 2 Cores | 50GB+ |
| Logstash | 1GB+ | 2GB+ | 1 Core | 10GB |
| Kibana | 1GB+ | 2GB+ | 1 Core | 1GB |
| Total System | 4GB | 8GB | 2+ Cores | 61GB+ |
From a technical perspective, the RAM requirement for Elasticsearch is paramount because it relies on the JVM heap. If the system possesses 4GB of RAM, it is standard practice to allocate half of the available memory to the JVM to prevent the operating system from swapping to disk, which would either crash the service or slow the indexing speed to a crawl. Consequently, while 4GB is the absolute minimum for a single-node "Elastic Stack server," 8GB is strongly recommended for production-grade stability.
The software environment must be strictly controlled. This guide focuses on Ubuntu 20.04, though the logic extends to 22.04 and 24.04. A critical administrative requirement is that all components of the stack must utilize the same version number. Installing Elasticsearch 8.x with Kibana 7.x, for example, will lead to API incompatibilities and failure in the communication layer.
Preliminary System Preparation and Java Runtime Environment
Before the installation of the Elastic binaries, the underlying operating system must be prepared. The Elastic Stack is built on Java, meaning the Java Runtime Environment (JRE) or Java Development Kit (JDK) is a non-negotiable prerequisite.
The installation process requires a user account with sudo permissions to manage system-level services and modify configuration files in /etc/. The initial phase involves updating the local package index to ensure the latest security patches are applied before introducing new software.
To install the required Java environment, the following commands are executed:
bash
sudo apt update
sudo apt install openjdk-17-jre-headless -y
The use of the headless version of the JRE is a technical optimization. Since the Elastic Stack server is typically a remote instance without a graphical user interface (GUI), the headless JRE removes unnecessary X11 libraries, reducing the attack surface and the disk footprint of the installation. After installation, the version is verified to ensure the JVM is active:
bash
java -version
Establishing the Elastic Repository and GPG Trust
The Elasticsearch components are not included in the default Ubuntu APT repositories. To obtain the official, signed binaries, the system must be pointed toward the Elastic artifacts repository. This process involves two distinct layers: establishing trust via a GPG key and adding the repository URL to the system's source list.
The GPG (GNU Privacy Guard) key is essential for security. It ensures that the packages downloaded from the Elastic servers have not been tampered with by a third party. This prevents "package spoofing," where a malicious actor could potentially inject a compromised version of the software into the update stream.
First, the GPG key is retrieved and stored in the shared keyrings directory:
bash
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
Once the key is in place, the repository is added to the APT sources list. For users deploying the 8.x version of the stack, the following command is used:
bash
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
Following the addition of the repository, the package cache must be refreshed:
bash
sudo apt update
Deploying and Configuring Elasticsearch
Elasticsearch is the central data store and search engine of the stack. It stores data as JSON documents and uses an inverted index to allow for lightning-fast searches across millions of log entries.
The installation is performed via the package manager:
bash
sudo apt install elasticsearch -y
Once installed, the configuration file located at /etc/elasticsearch/elasticsearch.yml must be modified to define how the node behaves within the network and how it manages data. The following configuration parameters are critical:
cluster.name: Defines the name of the cluster. This is used to group multiple nodes together. In a single-node setup, this is purely for identification.node.name: Assigns a unique name to this specific node (e.g.,node-1).network.host: Set tolocalhostfor security. If this were set to0.0.0.0, the database would be exposed to the public internet, which is a catastrophic security risk.discovery.type: Set tosingle-nodeto prevent the service from attempting to find other nodes in a cluster, which would otherwise lead to bootstrap errors.
The configuration file is edited using a text editor:
bash
sudo nano /etc/elasticsearch/elasticsearch.yml
The specific configuration entries should be as follows:
yaml
cluster.name: elk-cluster
node.name: node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: localhost
http.port: 9200
discovery.type: single-node
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
JVM Heap Optimization
One of the most common causes of Elasticsearch failure is an incorrectly configured JVM heap. By default, Elasticsearch may attempt to allocate more memory than the system can provide, leading to the process being killed by the Linux OOM Killer.
To manually set the heap size—typically to 50% of the total system RAM—the heap.options file is modified:
bash
sudo nano /etc/elasticsearch/jvm.options.d/heap.options
For a system with 4GB of RAM, the following settings are applied to reserve 2GB for the JVM:
text
-Xms2g
-Xmx2g
The -Xms flag sets the initial heap size, and -Xmx sets the maximum heap size. Setting these to the same value prevents the JVM from constantly resizing the heap, which improves performance stability.
Service Activation and Security Initialization
After configuration, the service must be enabled to start on boot and manually started for the first time:
bash
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch
To verify the status of the service:
bash
sudo systemctl status elasticsearch
Because xpack.security.enabled is set to true, the system requires authentication. The administrative password for the elastic user must be reset to establish a known credential:
bash
sudo /usr/share/elasticsearch/bin/elasticsearch-reset-password -u elastic
The generated password must be stored securely. To verify that Elasticsearch is running and responding to requests, a curl command is executed. Since SSL is enabled, the -k flag is used to ignore certificate warnings if using self-signed certificates:
bash
curl -k -u elastic:YOUR_PASSWORD https://localhost:9200
Deploying Kibana for Data Visualization
Kibana is the window into the data stored in Elasticsearch. It allows users to create visualizations, dashboards, and perform ad-hoc queries via a web browser.
The installation is performed via:
bash
sudo apt install kibana -y
Kibana configuration is handled in /etc/kibana/kibana.yml. By default, Kibana only listens on the localhost. To make it accessible to other machines or through a proxy, the server.host must be modified:
bash
sudo nano /etc/kibana/kibana.yml
The configuration should include:
yaml
server.port: 5601
server.host: "0.0.0.0"
server.name: "kibana-server"
By setting the host to 0.0.0.0, Kibana binds to all available network interfaces. However, this creates a security vulnerability if the server is exposed to the public internet.
Advanced Networking and Security with Nginx
To secure the Kibana interface and provide a professional URL, it is standard practice to use Nginx as a reverse proxy. Nginx sits between the user and Kibana, handling the incoming HTTP requests and forwarding them to port 5601.
This setup allows for the implementation of TLS/SSL certificates, ensuring that the sensitive log data being viewed in the browser is encrypted during transit. The following requirements must be met for a secure Nginx proxy:
- A Fully Qualified Domain Name (FQDN), such as
logs.your_domain.com. - DNS A-records pointing the domain and its
wwwsubdomain to the server's public IP address. - Nginx installed and configured with a server block.
For those seeking maximum security, the use of Let's Encrypt is strongly encouraged to obtain a free, trusted SSL certificate. This ensures that the communication between the administrator's browser and the Kibana interface is encrypted, preventing man-in-the-middle attacks on sensitive system telemetry.
Integrating Beats and Logstash for Data Ingestion
The final stage of the Elastic Stack is the data pipeline. The architecture follows a specific flow: Log Sources $\rightarrow$ Beats/Logstash $\rightarrow$ Elasticsearch $\rightarrow$ Kibana.
The Role of Filebeat
Filebeat is a lightweight shipper designed for logs. It monitors log files and forwards them to either Logstash or directly to Elasticsearch. Because it is written in Go, it has a very small memory footprint, making it ideal for installation on every single server in a network.
Filebeat simplifies the ingestion process by handling the "tailing" of files, ensuring that even if the server restarts, the shipper knows exactly where it left off in the log file.
The Role of Logstash
While Filebeat can send data directly to Elasticsearch, Logstash is used when the data needs transformation. Logstash acts as an ETL (Extract, Transform, Load) tool. It can:
- Parse unstructured logs into structured JSON.
- Filter out irrelevant data to save disk space.
- Enrich logs by adding metadata, such as geo-location based on IP addresses.
When a log entry is received by Logstash, it is passed through a pipeline of input, filter, and output plugins before being indexed into Elasticsearch.
Troubleshooting and Verification of Data Flow
After completing the installation and configuration of the entire stack, it is imperative to verify that logs are actually flowing through the system.
An administrator can verify this by searching the Elasticsearch index via the Kibana Dev Tools or via a direct API call. If the output of a search query shows 0 total hits, it indicates a failure in the pipeline. Common causes for this include:
- Filebeat not having read permissions for the target log files.
- Logstash failing to parse a specific log format, causing the packets to be dropped.
- Network firewall rules blocking traffic on ports 9200 (Elasticsearch) or 5044 (Logstash).
- Version mismatch between the Beats shipper and the Elasticsearch cluster.
Conclusion
The deployment of the Elastic Stack on Ubuntu 20.04 transforms a fragmented logging environment into a powerful, centralized intelligence hub. By meticulously configuring the JVM heap, establishing a secure GPG-verified repository, and implementing a reverse proxy through Nginx, an organization can achieve high-performance observability. The transition from manual log inspection to an automated, visualized pipeline allows for the correlation of events across multiple servers, which is the only viable way to manage the complexity of modern distributed architectures. The synergy between Elasticsearch's indexing speed, Logstash's transformation capabilities, and Kibana's visualization tools ensures that system failures are not just detected, but understood and resolved with surgical precision.