Orchestrating the LGTM Observability Stack via Podman Rootless Containerization

The modern landscape of software engineering demands a level of visibility that traditional monitoring tools can no longer sustain. As microservices architectures proliferate, the ability to correlate metrics, logs, and traces becomes a mission-critical capability for maintaining system uptime and performance. At the center of this observability revolution lies Grafana, the industry-leading open-scale platform designed to provide rich, interactive dashboards and complex alerting mechanisms. While many engineers are accustomed to deploying Grafana within the Docker ecosystem, a significant paradigm shift is occurring toward more secure, daemonless container engines like Podman.

Podman, which functions as a Pod Manager, offers a transformative approach to container orchestration on the local host. Unlike its predecessors, Podman operates without a central daemon, which eliminates a single point of failure and significantly reduces the attack surface of the host system. This is achieved primarily through rootless container execution, allowing users to run containers without requiring administrative privileges. The impact of this architectural choice is profound: it provides enhanced security isolation, ensuring that even if a containerized service like Grafana were compromised, the attacker would not gain root access to the underlying Linux kernel. This makes Podman-based deployments an ideal choice for development monitoring, staging environments, and even production-grade observability stacks where security is paramount.

Furthermore, Podman provides a bridge between local development and large-scale cloud orchestration. It is fully OCI-compliant, meaning it can consume images built by Buildah, Docker, or Podman itself. Its command-line interface is so compatible with Docker that users can simply alias the podman command to docker to maintain workflow continuity. Most importantly, Podman introduces the ability to manage Kubernetes-style pods—groups of containers that share a network namespace—using the same YAML deployment formats used in production Kubernetes clusters. This capability allows for a seamless transition from a single-machine setup to a multi-node cluster, effectively treating the local container engine as a miniature version of a production environment.

Foundational Host Preparation and Environment Setup

Before initiating the deployment of the Grafana LGTM (Loki, Grafana, Tempo, Mimir) stack, the underlying host operating system must be prepared to handle the container workloads and network requirements. The most stable environments for this deployment are RPM-based Linux distributions. Specifically, Red Hat Enterprise Linux (RHEL), CentOS, Rocky Linux, and AlmaLinux provide the native package management tools necessary for a seamless installation.

The initial phase of deployment requires ensuring that the host's package repository is synchronized and that all security patches are applied. This prevents conflicts between the container engine and outdated system libraries. The process begins with a full system update followed by a controlled reboot to ensure all kernel-level changes are active.

The following terminal commands are utilized for this initialization:

bash sudo dnf update -y && sudo reboot

Once the system has returned to an operational state, the installation of the Podman engine must be verified. Because Podman is a core component of the new deployment strategy, its presence is non-negotiable. On the aforementioned RPM-based systems, the installation is straightforward:

bash sudo dnf install -y podman

With the engine installed, the environment is now capable of managing OCI-compliant images and orchestrating multi-container pods. The following table outlines the core characteristics of the Podman engine compared to traditional daemon-based engines:

Deploying the Complete Grafana LGTM Stack via Kubernetes YAML

While individual containers can be managed using podman run commands, the true power of Podman lies in its ability to interpret Kubernetes-style YAML files. This allows for the simultaneous deployment of the entire LGTM stack—comprising Loki for logs, Grafana for visualization, Tempo for traces, and Mimir for metrics—as a single, cohesive unit. This method mimics the behavior of a production Kubernetes cluster, ensuring that the local setup is a faithful representation of the production environment.

The deployment process begins by cloning a pre-configured repository that contains the necessary orchestration logic. This repository provides the grafana-stack.yaml file, which defines the pod structure and container relationships.

The following steps must be executed to prepare the local directory:

Clone the specialized repository to the local machine.
Navigate into the directory to access the configuration files.

bash git clone httpshttps://github.com/CastawayEGR/grafana-stack-podman cd grafana-stack-podman

Once the directory is established, the deployment is triggered using the podman play kube command. This command instructs Podman to parse the YAML file and spin up the defined pods and containers according to the specified network and volume configurations.

bash podman play kube grafana-stack.yaml

After the command executes, the state of the containers must be verified. The podman ps command provides a real-time view of all active containers within the engine.

bash podman ps

A successful deployment will show the various components of the LGTM stack running as part of the orchestrated pod. However, even if the containers are running, the host's firewall may block external access to the monitoring ports. To ensure that Grafana and its backends (Loki, Tempo, Mimir) are accessible from other machines on the network, the firewall ports must be explicitly opened using firewall-capable.

The following ports must be permitted to allow for metrics ingestion, log streaming, and dashboard viewing:

bash sudo firewall-cmd --permanent --add-port={3000/tcp,3100/tcp,4317/tcp,4318/tcp,9095/tcp,9096/tcp,9097/tcp,9411/tcp,14268/tcp} sudo firewall-cmd --reload

The mapping of these ports is critical for the functionality of the stack:

Port	Service	Purpose
3000	Grafana	Primary web interface for dashboarding
3100	Loki	Log ingestion and querying endpoint
9009	Prometheus/Mimir	Metrics collection and storage
4317/4318	OTLP (gRPC/HTTP)	OpenTelemetry trace ingestion
9095/9096/9097	Mimir Components	Internal microservice communication

Once the firewall is reloaded, the Grafana interface is accessible via a web browser at http://{machine_ip}:3000. Upon the first login, the default credentials (admin/admin) can be used, though these should be updated immediately in a production setting. To complete the integration, the newly deployed backends must be manually configured as data sources within the Grafana UI using the following endpoints:

Prometheus: http://{machine_ip}:9009/prometheus
Loki: http://{machine_ip}:3100
Tempo: http://{machine_ip}:3200

Advanced Configuration: Persistence, Security, and Automation

A robust observability platform cannot exist solely in a transient state. If a container is restarted or updated, all dashboards, user permissions, and data source configurations must persist. In the Podman ecosystem, this is achieved through the use of named volumes and the :Z flag, which handles SELinux relabeling for volume permissions.

Implementing Persistent Storage

To prevent data loss, a named volume must be created and attached to the Grafana container. This ensures that the /var/lib/grafana directory—where all critical metadata resides—is stored on the host''s physical disk rather than within the container's writable layer.

The sequence for creating a persistent Grafana instance is as follows:

Create a dedicated volume for Grafana data.
Execute the container with the volume mapped to the internal Grafana path.

bash podman volume create grafana-data podman run -d \ --name grafana-persistent \ -p 3001:3000 \ -v grafana-data:/var/lib/grafana:Z \ docker.io/grafana/grafana:latest

The :Z suffix is a critical technical detail for users on RHEL, CentOS, or Fedora. It instructs Podman to automatically relabel the files with the correct SELinux context, preventing "Permission Denied" errors when the container attempts to write to the host-mounted volume.

Customizing Credentials and Environment Variables

For enterprise or highly secure environments, relying on default credentials is an unacceptable risk. Podman allows for the injection of configuration via environment variables, which can define custom administrative users, passwords, and server URLs. This method is particularly useful for automated CI/CD pipelines where credentials must be injected during the deployment phase.

The following command demonstrates a highly customized deployment:

bash podman volume create grafana-custom-data podman run -d \ --name grafana-custom \ -p 3002:3000 \ -e GF_SECURITY_ADMIN_USER=myadmin \ -e GF_SECURITY_ADMIN_PASSWORD=my-grafana-secret \ -e GF_USERS_ALLOW_SIGN_UP=false \ -e GF_SERVER_ROOT_URL=http://localhost:3002 \ -e GF_PLUGINS_PREINSTALL=grafana-clock-panel,grafana-simple-json-datasource \ -v grafana-custom-data:/var/lib/grafana:Z \ docker.io/grafana/grafana:latest

In this configuration, several security and functional layers are applied:
- The administrative username is changed from admin to myadmin.
- The password is set to a secure, unique string.
- User sign-ups are disabled (GF_USERS_ALLOW_SIGN_UP=false) to prevent unauthorized access.
- Specific plugins, such as the grafana-clock-panel, are pre-installed during the container startup, reducing the manual configuration required after the container is running.

To verify that these custom credentials have been correctly applied, the Grafana API can be queried directly from the terminal using curl. This is an essential step in automated testing to ensure the service is not only running but is also correctly configured.

bash curl -s -u myadmin:my-grafana-secret http://localhost:3002/api/org | python3 -m json.tool

Automated Provisioning of Data Sources and Dashboards

The final pillar of a professional observability setup is "Configuration as Code." Rather than manually clicking through the Grafana UI to add Prometheus or Elasticsearch, engineers should use provisioning files. These files allow for the automated setup of data sources and dashboards every time the stack is deployed.

This process involves creating a specific directory structure on the host and mounting it into the container. This ensures that even if the container is destroyed, the configuration remains intact and ready for the next deployment.

The steps for establishing a provisioning directory are:

Create the necessary directory hierarchy for datasources and dashboards.
Define the data source configuration in a YAML format.

bash mkdir -p ~/grafana-provisioning/datasources mkdir -p ~/grafana-provisioning/dashboards

A sample provisioning file for Prometheus and Elasticsearch can be created using the following command. This file specifies the URL, the access type (proxy), and the specific metadata required for the connection (such as the index for Elasticsearch).

bash cat > ~/grafana-provisioning/datasources/prometheus.yml <<'EOF' apiVersion: 1 datasources: - name: Prometheus type: prometheus access: proxy url: http://host.containers.internal:9090 isDefault: true editable: true jsonData: timeInterval: '15s' - name: Elasticsearch type: elasticsearch access: proxy url: http://host.containers.internal:9200 editable: true jsonData: index: 'app-logs-*' timeField: '@timestamp' EOF

The use of host.containers.internal is a critical technical nuance. In a rootless Podman environment, containers cannot easily resolve the host's IP address. This special DNS name allows the containerized Grafana instance to reach services running on the host or in other containers within the same pod.

To run the final, fully automated version of Grafana, the container must be launched with these provisioning files mounted as volumes:

bash podman volume create grafana-provisioned-data podman run -d \ --name grafana-provisioned \ -p 3003:3000 \ -v ~/grafana-provisioning/datasources:/etc/grafana/provisioning/datasources:Z \ -v ~/grafana-provisioning/dashboards:/etc/grafana/provisioning/dashboards:Z \ docker.io/grafana/grafana:latest

Analytical Conclusion

The deployment of the Grafana LGTM stack using Podman represents a significant advancement in the maturity of local observability workflows. By moving away from the centralized, high-privilege architecture of Docker and toward the daemonless, rootless model of Podman, engineers can achieve a level of security and operational fidelity that was previously difficult to maintain outside of a full Kubernetes cluster.

The integration of Kubernetes-style pods via podman play kube provides a critical bridge for DevOps professionals, allowing for the exact replication of production-level networking and service discovery on a single workstation. This reduces the "it works on my machine" phenomenon and ensures that the complex interactions between Loki, Mimir, Tempo, and Grafana are tested under near-identical conditions to the production environment.

Furthermore, the transition to "Configuration as Code" through environment variables and provisioning files transforms the observability stack from a manual, error-prone setup into a repeatable, version-controlled asset. The ability to automate data source registration and plugin installation via Podman volumes and environment injections is essential for the modern era of GitOps. Ultimately, this architecture provides a scalable, secure, and highly efficient foundation for monitoring the next generation of distributed, microservice-based applications.