The pursuit of deep observability within an enterprise-grade Kubernetes ecosystem necessitates a sophisticated approach to metrics collection and visualization, particularly when operating within the constraints of OpenShift 4. While Red Hat provides a robust, prepackaged monitoring stack within the opens/openshift-monitoring namespace, this specific implementation is architected as a read-only environment. For engineers and architects tasked with customizing dashboards, injecting new data sources, or creating bespoke visualization layers, the default stack presents a significant barrier due to its immutable nature. To circumvent these limitations without compromising the underlying stability of the cluster, a parallel, self-supported observability architecture must be deployed. This architecture leverages the community-supported Grafana Operator and a customized Prometheus configuration to allow for extensible, high-fidelity monitoring of workloads, including specialized use cases like OpenShift Virtualization or the Authentication Hub (SSP) services. By utilizing a GitOps-driven approach, specifically through OpenShift GitOps (ArgoCD), administrators can manage these observability resources as code, ensuring that the monitoring layer is as scalable, reproducible, and resilient as the applications it tracks.
The Architectural Dilemma of Read-Only Monitoring Namespaces
The foundational challenge in OpenShift observability lies in the structural design of the openshift-monitoring namespace. This namespace is managed by the cluster's internal Prometheus Operator, and its configurations are essentially locked to prevent unauthorized modifications that could destabilize the cluster's core health metrics.
The primary impact of this read-only design is the inability to import custom dashboards or modify existing alert rules directly within the default Grafana instance. For a DevOps engineer, this means that any attempt to enhance visibility into application-specific metrics—such as billing metrics for O/CP Virtualization or performance data for Authentication Hub—is met with a lack of persistence for any changes made via the UI.
To resolve this, architects must move beyond the default namespace. The strategy involves deploying a separate, user-controlled instance of Grafana using the community-supported Grafana Operator sourced from OperatorHub. This approach provides several critical advantages and architectural considerations:
- The existing Prometheus Operator remains untouched and in a supported state by Red Hat, preserving the integrity of the cluster's core monitoring.
- A new, dedicated project (for example,
my-grafanaorgrafana) can be established to host the custom Grafana resources. - The deployment of a self-supported Grafana instance allows for the full utilization of the Grafana Operator's capabilities, including the management of dashboards, datasources, and folders as Kubernetes Custom Resources (CRDs).
It is imperative to recognize that because these operators are sourced from OperatorHub and are community-provided, they have not been vetted or verified by Red Hat. The stability of these community operators is technically unknown from a Red Hat support perspective, and their use carries an inherent risk that must be managed through rigorous testing and a GitOps-driven deployment pipeline.
Deployment Strategies via OpenShift GitOps and ArgoCD
Manual intervention in the deployment of observability stacks is an anti-pattern in modern cloud-native environments. To achieve high availability and configuration consistency, the deployment of the custom Grafana instance should be orchestrated using OpenShift GitOps, which utilizes ArgoCD to synchronize the desired state defined in a Git repository with the actual state of the cluster.
The deployment process begins with the installation of the OpenShift GitOps operator from OperatorHub. Once installed, the application can be deployed into the opens and openshift-gitops namespace. A highly efficient method to apply the necessary configurations is through the use of the oc command-line interface to apply pre-defined application manifests.
The following command demonstrates the application of a custom Grafana application definition:
oc apply -f custom-grafana.application.yaml
This GitOps approach ensures that every change to the Grafana configuration—whether it be a change in the Grafana version, a new datasource, or a modified dashboard—is version-controlled and auditable. This is particularly critical when deploying in disconnected or air-gapped environments where manual configuration changes are difficult to replicate and verify.
Implementing the Community Grafana Operator
The Grafana Operator serves as the control plane for managing Grafana instances and their associated resources. It automates the synchronization of Kubernetes custom resources with the actual configuration of the Grafana instance, allowing for the management of data sources, dashboards, and folders as native Kubernetes objects.
Installation Procedures
There are two primary methodologies for installing the Grafana Operator: using Helm for a standard Kubernetes-style deployment, or using the OpenShift Web Console via OperatorHub for an OpenShift-native experience.
For users preferring a programmatic approach via the command line, Helm can be utilized as follows:
Add the official Grafana Helm repository to your local configuration:
helm repo add grafana https://grafana.github.io/helm-chartsUpgrade or install the operator into your cluster:
helm upgrade -i grafana-operator grafana/grafana-operator
For administrators utilizing the OpenShift Web Console, the process is more visual but follows a similar logic:
- Navigate to the "Operators" menu and select "OperatorHub".
- Search for "Grafana" in the search bar.
- Select the Grafana operator from the results.
- A warning will appear, notifying you that this operator is not vetted by Red Hat. Click "Continue" to proceed.
- Configure the installation parameters, accepting the defaults unless specific customization is required.
- Click "Install".
- If the operator is not configured for automatic updates, you may see a status indicating that manual approval is required. In this event, navigate to "Installed Operators", select the Grafana Operator, and click "Approve".
Provisioning Grafana Instances
Once the operator is installed and running, the creation of the actual Grafana instance is managed through the creation of a Custom Resource (CR). This can be performed via the OpenShift UI or via kubectl.
Using the Web Console:
- Navigate to "Installed Operators".
- Locate the "Grafana Operator".
- Find the "Grafana" section under the operator's managed resources.
- Click "Create Grafana".
- Configure the resource specifications (e.g., storage requirements, service type) and click "Create".
Upon successful creation, the operator automatically manages the deployment of the Grafana Pod and creates a dedicated service account, such as grafana-sa, which is essential for the Grafana instance to interact with other cluster resources securely.
Advanced Configuration: Security Contexts and Persistent Storage
In highly regulated or high-security environments, the configuration of the Grafana Pod's security context is paramount. When deploying via Helm or custom manifests, administrators must often define specific user IDs (UID) and group IDs (GID) to comply with Pod Security Standards (PSS) or OpenShift's stringent Security Context Constraints (SCC).
The following configuration fragment illustrates how to set the runAsUser and fsGroup parameters to ensure the Grafanam container operates with the correct permissions:
--set grafana.containerSecurityContext.runAsUser=${GRAFANA_UID} \
--set grafana.podSecurityContext.runAsUser=${GRAFANA_UID} \
--set grafana.podSecurityContext.fsGroup=${GRAFANA_FSGROUP} \
--version=2.7.3
Furthermore, the reliability of the observability stack is directly linked to the use of Persistent Volumes (PVs). While both the Prometheus Operator and Grafana can technically function using ephemeral storage, this is unsuitable for production environments. Without persistent volumes, any pod restart or node failure results in the total loss of historical metrics, dashboards, and user configurations. For high-availability deployments, administrators should configure PersistentVolumeClaims (PVCs) for both the Prometheus instance and the Grafana instance to ensure data durability and availability.
Integrating Prometheus via Grafana Data Source Objects
A critical component of this architecture is the ability of the custom Grafana instance to scrape metrics from the existing OpenShift Prometheus deployment. This requires the creation of a GrafanaDataSource object, which tells Grafana how to communicate with the Prometheus service and how to authenticate the request.
Token Acquisition and Authentication
Because the Prometheus instance resides in a restricted namespace, Grafana must present a valid bearer token to access the metrics. This token is typically retrieved from the openshift-user-workload-monitoring secret. The following shell commands demonstrate the extraction of this token:
SECRET_NAME=kubectl get secret -n openshift-user-workload-monitoring | grep prometheus-user-workload-token | head -n 1 | awk '{print $1 }'`
TOKEN=echo $(kubectl get secret $SECRET_NAME -n openshift-user-workload-monitoring -o json | jq -r '.data.token') | base64 -d``
Configuring the Datasource Resource
Once the token is secured, the administrator must apply a GrafanaDataSource custom resource. This resource defines the connection string to the Prometheus service, the use of TLS, and the injection of the authorization header containing the bearer token.
The following manifest must be applied to the monitoring namespace:
yaml
apiVersion: integreatly.org/v1alpha1
kind: GrafanaDataSource
metadata:
name: grafana-datasource
namespace: monitoring
spec:
name: grafana-datasource.yaml
datasources:
- name: Prometheus
type: prometheus
version: 1
access: proxy
editable: true
isDefault: true
url: 'https://prometheus-k8s.openshift-monitoring.svc:9091'
jsonData:
timeInterval: 5s
tlsSkipVerify: true
httpHeaderName1: 'Authorization'
secureJsonData:
httpHeaderValue1: "Bearer ${TOKEN}"
After applying this configuration, the Grafana Pod must be restarted to ensure the new configuration is loaded into the running instance:
kubectl delete pod -l app=grafana -n monitoring
Network Exposure and Service Accessibility
To access the Grafana dashboard from external networks or local development machines, the Grafana Service must be exposed via a Route (OpenShift Ingress) or a LoadBalancer.
Resolving the Grafana FQDN
The Fully Qualified Domain Name (FQDN) of the Grafana service must be resolvable to the Ingress IP Address. In enterprise environments, this involves updating Cloud DNS or local DNS records.
The FQDL typically follows the pattern:
grafana.<DOMAIN>
If the administrator does not have control over the DNS infrastructure, the /etc/hosts file on the local machine can be modified to map the Ingress IP to the service FQDN:
<openshift-ingress-ip> grafana.<DOMAIN>
Retrieving Ingress Information
To identify the correct IP address or route for exposure, administrators can use the following commands depending on the type of controller being used:
For the NGINX Ingress Controller:
kubectl get ingress -n monitoring
For the OpenShift Ingress Controller (Route-based):
kubectl get svc -n openshift-ingress
The IP address can be extracted from the "Address" column of the Ingress object or the "EXTERNAL IP" column of the service, respectively.
Data Source and Metric Utilization
A fully configured Grafana instance in OpenShift enables the visualization of highly specialized data. For instance, in an OpenShift Virtualization environment, administrators can create customized billing dashboards. These dashboards can track resource consumption and costs, which is vital for organizations that prefer not to transmit their sensitive performance data to external consoles like console.redhat.com via Red Hat Insights, especially in air-gapped or disconnected environments.
The ability to use the Grafana Operator to manage folders, dashboards, and data sources "as code" allows for a seamless transition of observability configurations between development, staging, and production clusters. By treating the observability layer as a standard part of the application lifecycle, organizations can ensure that every deployment is accompanied by the necessary monitoring, alerting, and visibility tools required for operational excellence.
Analysis of Observability Architectures
The implementation of a community-driven Grafana and Prometheus stack on OpenShift 4 represents a strategic trade-off between the "out-of-the-box" stability of Red Hat's managed services and the operational flexibility required by advanced engineering teams. The primary strength of this architecture lies in its extensibility; by bypassing the read-only constraints of the default openshift-monitoring namespace, teams can implement custom-tailored monitoring solutions that are critical for complex workloads such as OpenShift Virtualization or highly regulated Authentication Hub services.
However, this flexibility introduces a significant responsibility regarding lifecycle management. The reliance on Community Operators means that the burden of testing, upgrading, and securing the observability stack shifts entirely to the cluster administrator. The use of GitOps (ArgoCD) is not merely a recommendation but a requirement for managing this complexity. Without a GitOps-driven approach, the configuration of the Grafana instance—specifically the delicate management of bearer tokens for Prometheus authentication and the orchestration of security contexts—becomes prone to configuration drift and manual error.
Ultimately, the success of this deployment hinges on the integration of the Grafana Operator with a robust automation pipeline. When executed correctly, this architecture provides a highly scalable, auditable, and deep-drilling observability layer that complements the core cluster monitoring while providing the granular visibility necessary for modern, cloud-native operations.