Deployment and Configuration of the Elastic Stack on CentOS 7 for Enterprise Log Management

The Elastic Stack, commonly referred to as the ELK stack, represents a sophisticated integration of three open-source products: Elasticsearch, Logstash, and Kibana. When deployed on a CentOS 7 environment, this stack transforms a standard Linux server into a powerful centralized hub for search, analysis, and visualization of massive datasets. The primary utility of this architecture lies in its ability to ingest unstructured log data from multiple disparate sources, normalize that data into a searchable format, and provide a graphical interface for real-time monitoring. For system administrators and DevOps engineers, the ELK stack eliminates the tedious manual process of scouring individual log files across multiple servers, replacing it with a unified dashboard that can identify anomalies, security breaches, or performance bottlenecks in seconds.

The architecture is fundamentally based on a pipeline: data is collected and processed by Logstash (or Beats), stored and indexed by Elasticsearch, and visualized by Kibana. In a CentOS 7 deployment, this process leverages the stability of the RHEL-based ecosystem, utilizing the YUM package manager for dependency resolution and systemd for service management. Whether deploying on a single machine for small-scale monitoring or as a foundation for a distributed cluster, the CentOS 7 platform provides the necessary kernel stability and resource management required to handle the high I/O and memory demands of the Elastic ecosystem.

Architectural Components of the ELK Stack

The ELK stack is not a single application but a collection of three distinct yet interdependent tools. Understanding the specific role of each component is critical for successful deployment and troubleshooting.

Elasticsearch serves as the core of the entire stack. It is a distributed RESTful search and analytics NoSQL engine built upon Apache Lucene. Unlike relational databases, Elasticsearch is designed for near real-time search capabilities and can scale horizontally to handle petabytes of data. It functions by indexing documents in a JSON-like format, allowing for complex queries to be executed with extreme efficiency.

Logstash acts as the server-side data processing pipeline. Its primary responsibility is to ingest data from multiple sources simultaneously, transform it through a series of filters, and then send it to a destination, most commonly Elasticsearch. It functions as the "transformer" of the stack, ensuring that raw log strings are converted into structured data that Elasticsearch can index.

Kibana provides the Graphical User Interface (GUI) that sits on top of Elasticsearch. It is a web application that allows users to visualize the data stored in Elasticsearch using charts, graphs, and maps. Through Kibana, administrators can create dashboards to monitor system health, track invalid SSH login attempts, or analyze application error rates in real-time.

Hardware and Software Prerequisites

Before initiating the installation process on CentOS 7, the system must meet specific minimum requirements to ensure the stability of the Java Virtual Machine (JVM) and the responsiveness of the search engine.

The operating system must be CentOS 7, and the user must possess full root access to perform package installations and modify system configuration files. Because the ELK stack is resource-intensive, particularly regarding memory and CPU, the following specifications are recommended:

OS: CentOS 7
RAM: 4GB (Minimum)
CPU: 2 Cores (Minimum)

The actual resource requirements may scale upward depending on the volume of logs being gathered. High-velocity log environments will require significantly more RAM to prevent the JVM from triggering OutOfMemory (OOM) errors.

A critical software prerequisite is the Java Development Kit (JDK). Elasticsearch is written in Java and requires a compatible JVM to execute. While the stack supports both OpenJDK and Oracle Java, there is a specific recommendation for Oracle JDK 1.8 in certain deployment scenarios to ensure maximum compatibility and performance.

Java Installation and Environment Setup

The first operational step in deploying the ELK stack is the installation of the Java runtime. Without a functional JDK, the Elasticsearch service will fail to initialize.

For those opting for OpenJDK, the installation can be performed using the YUM package manager. This ensures that all necessary dependencies are resolved automatically.

sudo yum install java-1.8.0-openjdk

Alternatively, for a more comprehensive development environment, the following command installs both the runtime and the development kit:

sudo yum -y install java-openjdk-devel java-openjdk

Following the installation, it is mandatory to verify that the Java environment is correctly mapped to the system path. This is achieved by checking the version:

java -version

An example of a successful installation output for a more recent version (such as OpenJDK 17 on related Fedora/CentOS systems) would appear as:

openjdk version "17.0.5" 2022-10-18
OpenJDK Runtime Environment (Red_Hat-17.0.5.0.8-1.fc37) (build 17.0.5+8)
OpenJDK 64-Bit Server VM (Red_Hat-17.0.5.0.8-1.fc37) (build 17.0.5+8, mixed mode, sharing)

Repository Configuration for the Elastic Stack

The ELK components are not available in the default CentOS 7 base repositories. Therefore, a dedicated Elastic repository must be added to the system to allow YUM to fetch the official packages. The configuration differs based on whether the administrator intends to deploy the 7.x or 8.x series of Elasticsearch.

For those deploying Elasticsearch 8.x, the following configuration must be written to the repository file:

cat <<EOF | sudo tee /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-8.x]
name=Elasticsearch repository for 8.x packages
baseurl=https://artifacts.elastic.co/packages/8.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

For the 7.x series, a similar process is used, though the base URL points to the 7.x artifacts. This repository setup ensures that the system can perform updates and security patches directly from the Elastic official servers.

Elasticsearch Installation and Verification

Once the repository is active, Elasticsearch can be installed. This component serves as the database for the entire stack. After installation, the service is typically managed via systemd, with the configuration file located at /usr/lib/systemd/system/elasticsearch.service.

To verify that Elasticsearch is running and accessible, administrators can use the curl command to query the API. The method of verification depends on the version installed.

For Elasticsearch 8, which implements stricter security defaults, the query requires a certificate and authentication:

sudo curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic https://localhost:9200

A successful response will return a JSON object containing the cluster details:

json { "name" : "cent7.novalocal", "cluster_name" : "elasticsearch", "cluster_uuid" : "5GFmuAkwQ4Sxrrrg4G-b6A", "version" : { "number" : "8.2.0", "build_flavor" : "default", "build_type" : "rpm", "build_hash" : "b174af62e8dd9f4ac4d25875e9381ffe2b9282c5", "build_date" : "2022-04-20T10:35:10.180408517Z", "build_snapshot" : false, "lucene_version" : "9.1.0", "minimum_wire_compatibility_version" : "7.17.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "You Know, for Search" }

For Elasticsearch 7 and below, the verification is a simpler HTTP request:

curl http://127.0.0.1:9200

To ensure that the indexing engine is fully operational, it is recommended to create a test index using a PUT request:

curl -X PUT "http://127.0.0.1:9200/mytest_index"

The expected response is {"acknowledged":true,"shards_acknowledged":true,"index":"mytest_index"}, confirming that the cluster can successfully write data.

Logstash and Kibana Integration

With Elasticsearch operational, the pipeline is completed by installing Logstash and Kibana. Logstash handles the ingestion and parsing, while Kibana provides the visual interface.

Kibana is installed via the same YUM repository used for Elasticsearch. Once installed, the administrator must log into the Kibana dashboard using the admin credentials. This interface allows for the creation of indices and the management of data streams.

A common workflow involves the use of "Beats," which are lightweight data shippers. For example, Filebeat can be installed on client machines (such as a client at 192.168.0.70 sending data to a server at 192.168.0.180) to ship logs to the ELK stack.

Within the Kibana dashboard, users can:

Create default indices such as filebeat-* to categorize incoming data.
Mark specific beats as favorites using the star button for quicker access.
Use the 'Discover' menu to view raw log files from multiple client servers.
Analyze JSON output, such as logs indicating invalid SSH login attempts from client machines.

Technical Specifications Comparison

The following table summarizes the roles and technical characteristics of the components within the ELK stack deployment.

Component	Role	Technology Base	Primary Function	Interface
Elasticsearch	Storage & Search	Lucene / NoSQL	Indexing and Querying	REST API
Logstash	Processing	Java	Ingestion and Parsing	Pipeline/CLI
Kibana	Visualization	JavaScript/Web	Dashboards and Charts	Web Browser
Filebeat	Shipping	Go	Lightweight Log Transport	Agent/Daemon

Analysis of Deployment Impact and Logic

The deployment of ELK on CentOS 7 creates a highly scalable environment for log management. The logic of this setup relies on the separation of concerns: the client machine (using Filebeat) does not perform any heavy processing, which prevents the monitoring process from consuming the resources of the production application. Instead, the heavy lifting of parsing is shifted to Logstash and the heavy lifting of indexing is shifted to Elasticsearch.

The impact on a system administrator is profound. Instead of logging into ten different servers to search for a specific error string across ten different /var/log/messages files, the administrator can perform a single query in Kibana. This reduces the Mean Time to Resolution (MTTR) for system outages and security incidents. Furthermore, the use of JSON as the primary data format allows for the integration of diverse log types—from web server access logs to kernel panic messages—into a single, unified timeline.

The decision to use CentOS 7 as the host OS is rooted in the need for a stable, long-term support (LTS) environment. Since the Elastic Stack requires significant memory allocation (often requiring adjustments to the jvm.options file to increase heap size), the predictable memory management of CentOS 7 ensures that the system remains stable under high load.

Conclusion

The successful installation of the ELK stack on CentOS 7 provides an enterprise-grade foundation for observability and security auditing. By integrating Elasticsearch's powerful indexing, Logstash's flexible processing pipelines, and Kibana's intuitive visualization, organizations can move from a reactive logging posture to a proactive monitoring strategy. The requirement for Java 8 (or newer) and a minimum of 4GB of RAM underlines the resource-heavy nature of the stack, but the resulting ability to analyze complex data patterns across multiple clients justifies the investment.

The transition from raw logs to actionable insights is facilitated by the seamless flow of data from a client-side Beat agent to the centralized ELK server. Whether the goal is to monitor for invalid SSH logins or to track application performance, the combination of a RHEL-based operating system and the Elastic suite provides the reliability and scalability necessary for modern infrastructure management. The ability to verify the installation through REST API calls and manage the environment via systemd ensures that the stack is not only functional but also maintainable over the long term.