Centralized Log Management via Elastic Stack Deployment on CentOS 7

The implementation of the Elastic Stack, formerly known as the ELK Stack, on a CentOS 7 environment represents a sophisticated approach to centralized logging, offering an integrated ecosystem for searching, analyzing, and visualizing logs from disparate sources. At its core, centralized logging is the practice of aggregating logs generated from any source in any format into a single, searchable repository. This architectural strategy is critical for modern IT infrastructure, as it eliminates the need for administrators to manually log into individual servers to troubleshoot issues. By correlating logs from multiple servers within a specific time frame, organizations can identify systemic issues that span across a distributed environment, which would otherwise remain hidden in isolated log files.

The Elastic Stack comprises four primary components that function in a pipeline: Elasticsearch, Logstash, Kibana, and Beats (specifically Filebeat). Elasticsearch serves as the heart of the stack, acting as a distributed search and analytics engine that stores the logs sent by clients. Logstash functions as the processing engine, transforming and filtering the data before it reaches the storage layer. Kibana provides the visualization layer, offering a web-based interface to inspect and analyze the data stored in Elasticsearch. Finally, Filebeat acts as the log shipping agent, residing on client machines to forward and centralize logs and files.

Deploying this stack on CentOS 7 requires a meticulous approach to versioning and resource allocation. A fundamental requirement of the Elastic Stack is version parity; it is mandatory to use the same version across the entire stack—Elasticsearch, Logstash, and Kibana—to ensure compatibility and prevent data corruption or communication failures between the components. Furthermore, the hardware requirements for the stack are highly variable. RAM allocation must be carefully planned based on the volume of data being ingested, the complexity of the queries being executed, and the overall size of the deployment environment.

Core Components and Functional Architecture

The Elastic Stack is designed as a data pipeline where information flows from the source to the visualization layer. Each component plays a distinct role in ensuring that raw log data is converted into actionable insights.

  • Elasticsearch: This component is responsible for storing the logs that are sent by the clients. It is a NoSQL database that allows for near real-time search and analysis of massive datasets.
  • Logstash: This is the processing layer. Logstash gathers data from various sources, transforms it through filters, and sends it to the storage layer.
  • Kibana: This component provides the web interface. It allows users to create dashboards, visualize trends, and perform deep-dive inspections of the logs.
  • Filebeat: This is a lightweight "Beat" used for forwarding and centralizing logs. It is installed on client machines to ship logs to the central server.

The interaction between these components creates a robust telemetry system. For example, in a typical production environment, Filebeat monitors a log file on a Fedora or Debian client machine and ships it to Logstash. Logstash then parses the raw text into structured JSON and pushes it into Elasticsearch. The administrator then accesses Kibana via a web browser to query the data. Because Kibana is typically only available on the localhost by default, it is common practice to use Nginx as a reverse proxy to make the interface accessible over a network.

Manual Installation Procedures on CentOS 7

The manual installation of the Elastic Stack on CentOS 7 involves setting up the official Elastic repositories and managing the services via systemd. A baseline requirement for this process is the installation of the Java OpenJDK, as the stack is built on Java.

Elasticsearch Installation

The first step in the deployment is the installation of the search engine. This requires importing the GPG key to ensure package integrity and configuring the YUM repository.

  1. Install Java and import the GPG key:
    yum install java-openjdk -y
    rpm --import http://packages.elastic.co/GPG-KEY-elasticsearch

  2. Create the repository file:
    cat <<EOF > /etc/yum.repos.d/elasticsearch.repo
    [elasticsearch-2.x]
    name=Elasticsearch repository for 2.x packages
    baseurl=http://packages.elastic.co/elasticsearch/2.x/centos
    gpgcheck=1
    gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
    enabled=1
    EOF

  3. Execute the installation and enable the service:
    yum install elasticsearch -y
    systemctl enable elasticsearch
    systemctl start elasticsearch

Kibana Installation

Kibana must be installed using a version that matches the Elasticsearch installation to avoid API mismatches.

  1. Create the repository file:
    cat <<EOF > /etc/yum.repos.d/kibana.repo
    [kibana-4.4]
    name=Kibana repository for 4.4.x packages
    baseurl=http://packages.elastic.co/kibana/4.4/centos
    gpgcheck=1
    gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
    enabled=1
    EOF

  2. Execute the installation and enable the service:
    yum install kibana -y
    systemctl enable kibana
    systemctl start kibana

Logstash Installation

Logstash completes the core triad by providing the ingestion and processing capabilities.

  1. Create the repository file:
    cat <<EOF > /etc/yum.repos.d/logstash.repo
    [logstash-2.2]
    name=logstash repository for 2.2 packages
    baseurl=http://packages.elasticsearch.org/logstash/2.2/centos
    gpgcheck=1
    gpgkey=http://packages.elasticsearch.org/GPG-KEY-elasticsearch
    enabled=1
    EOF

  2. Execute the installation and enable the service:
    yum install logstash -y
    systemctl enable logstash
    systemctl start logstash

Automated Deployment via Ansible

For organizations managing multiple servers, manual installation is inefficient. Ansible, an open-source automation tool, allows for the programmatic deployment of the Elastic Stack across a cluster of CentOS 7 servers. This approach ensures consistency and reduces the risk of human error during configuration.

Inventory and Architecture

The automation process begins with the creation of an Ansible inventory file. This file contains the list of CentOS 7 servers that will form the Elasticsearch cluster, identified by their IP addresses or hostnames. This allows the administrator to target specific groups of servers, such as the master-elk-server or the kibana-server.

Kibana Automated Configuration

The automated installation of Kibana involves downloading the RPM and managing enrollment tokens for secure communication.

  • Download and Install:
    The Ansible playbook uses the get_url module to retrieve the RPM and the yum module to install it:
    ```yaml
  • name: Download Kibana RPM
    geturl:
    url: https://artifacts.elastic.co/downloads/kibana/kibana-{{ elk
    version }}-x8664.rpm
    dest: /tmp/kibana.rpm
    when: inventory
    hostname in groups["kibana-server"]

  • name: Install kibana RPM
    yum:
    name: /tmp/kibana.rpm
    state: present
    when: inventory_hostname in groups["kibana-server"]
    ```

  • Enrollment Token Management:
    For secure setup, an enrollment token must be generated on the master server and distributed to the Kibana nodes:
    ```yaml

  • name: Generate enrollment token for Kibana nodes.
    shell: /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
    register: kibanatokennode
    when: inventory_hostname in groups["master-elk-server"]

  • name: Keep Kibana tokens on different servers.
    copy:
    content: "{{ hostvars[groups['kibana-server'][0]]['kibanatokennode']['stdout'] }}"
    dest: /etc/kibana/kibanatokennode.txt
    loop: "{{ groups['kibana-server'] }}"
    when:

    • inventory_hostname in groups["kibana-server"]
    • inventoryhostname != item
      delegate
      to: "{{ item }}"
      ```
  • Finalizing Enrollment:
    The Kibana setup is finalized by running the enrollment command using the registered token:
    ```yaml

  • name: Register/Enrolling kibana
    shell: /usr/share/kibana/bin/kibana-setup --enrollment-token {{ kibanatoken.stdout }}
    when: inventory
    hostname in groups["kibana-server"]
    ```

Configuration Tuning and Security

The Ansible playbooks also handle the modification of the kibana.yml configuration file. This includes removing default host settings to allow for custom configurations and inserting specific blocks of configuration using the blockinfile module.

  • Configuration Modification:
    ```yaml
  • name: Delete line elasticsearch.hosts
    lineinfile:
    path: /etc/kibana/kibana.yml
    regexp: '^elasticsearch.hosts:'
    line: '#'
    when: inventory_hostname in groups["kibana-server"]

  • name: Config Kibana
    blockinfile:
    path: /etc/kibana/kibana.yml
    block: "{{ kibanaconfig }}"
    marker: "# {mark} Kibana configuration"
    backup: yes
    when: inventory
    hostname in groups["kibana-server"]
    ```

For environments requiring enhanced security, the stack can be configured with SSL/TLS. This can be achieved by generating self-signed certificates using the elasticsearch-certutil tool:
/usr/share/elasticsearch/bin/elasticsearch-certutil cert -name kibana-server --self-signed

Deployment Specifications and Testing Environments

When designing a test environment for the Elastic Stack, it is essential to simulate a real-world distributed architecture. A typical testing setup consists of a central server and multiple client machines running different Linux distributions to verify cross-platform compatibility.

Environment Configuration Table

Machine Role Operating System IP Address Primary Function
Central Server RHEL 192.168.100.247 Runs Elasticsearch, Logstash, Kibana
Client Machine #1 Fedora 192.168.100.133 Runs Filebeat; ships logs to central server
Client Machine #2 Debian 192.168.0.134 Runs Filebeat; ships logs to central server

Technical Impact of Resource Constraints

The performance of the ELK Stack is directly tied to the underlying hardware. Because Elasticsearch indexes data in real-time and Kibana performs complex aggregations, RAM is the most critical resource. If the RAM is insufficient for the volume of data, the Java Virtual Machine (JVM) may experience frequent Garbage Collection (GC) pauses, leading to increased latency in log searches or even service crashes. Consequently, administrators must scale memory based on the complexity of the queries and the volume of logs processed by Logstash.

Security Hardening and Network Integration

Implementing the Elastic Stack without security measures exposes the data to unauthorized access, especially since Kibana provides a direct window into the system logs.

  • TLS Support: The use of OpenSSL is recommended to encrypt the traffic between the components. This prevents "man-in-the-middle" attacks where log data could be intercepted during transmission from Filebeat to Logstash.
  • Reverse Proxying with Nginx: By default, Kibana binds to the localhost. To make it accessible to a broader group of analysts without exposing the Kibana port directly to the internet, Nginx is used. Nginx acts as the gateway, forwarding web requests to the Kibana service while providing an additional layer of security and the ability to implement SSL termination.
  • GPG Verification: The use of rpm --import for the Elastic GPG keys ensures that the software being installed is authentic and has not been tampered with, which is a critical step in maintaining the security chain of the server.

Detailed Component Interaction Analysis

The synergy between the components of the Elastic Stack creates a cohesive data pipeline that transforms raw, unstructured data into a searchable index.

  • The Ingestion Phase: Filebeat, as the light-weight agent, monitors specific log files (e.g., /var/log/syslog or application logs). It avoids consuming excessive CPU or RAM, ensuring that the client machine's primary application is not impacted.
  • The Transformation Phase: Logstash receives the data from Filebeat. Here, the "Grok" filter is often used to parse unstructured text into structured fields. For example, a standard Apache log line is broken down into "IP Address", "Timestamp", "HTTP Method", and "Response Code".
  • The Indexing Phase: Elasticsearch receives the structured JSON from Logstash. It indexes the data, making every field searchable. The distributed nature of Elasticsearch allows this data to be spread across multiple nodes if the cluster expands.
  • The Visualization Phase: Kibana queries the Elasticsearch API to retrieve data. It allows users to create "Discover" views for raw log inspection or "Dashboard" views for high-level trends, such as the number of 404 errors per hour across a fleet of servers.

Conclusion

The deployment of the Elastic Stack on CentOS 7 provides a powerful mechanism for centralized logging and observability. By integrating Elasticsearch for storage, Logstash for processing, Kibana for visualization, and Filebeat for shipping, organizations can transition from a reactive troubleshooting posture to a proactive monitoring strategy. The use of automation tools like Ansible further enhances this process, allowing for the rapid scaling of the stack while maintaining strict version parity and configuration consistency across the environment.

The technical success of such a deployment depends on the careful alignment of hardware resources, particularly RAM, and the implementation of security measures such as TLS and Nginx proxying. Whether installed manually through YUM repositories or orchestrated via Ansible playbooks, the Elastic Stack transforms the daunting task of managing logs across a heterogeneous environment—comprising RHEL, Fedora, and Debian—into a streamlined, efficient, and highly searchable operation. The ability to correlate logs across multiple servers within a specific timeframe remains the single most valuable outcome of this architecture, enabling the rapid identification of systemic failures in complex IT infrastructures.

Sources

  1. CyberRanges
  2. LinkedIn - Sirisoft PCL
  3. DigitalOcean
  4. Timothy Quinn
  5. TecMint

Related Posts