OpenSearch Kubernetes Operator Orchestration

The deployment of OpenSearch within a Kubernetes environment represents a shift toward automated lifecycle management, moving away from manual pod configuration toward a declarative state. Central to this evolution is the Kubernetes OpenSearch Operator, a specialized controller designed to automate the deployment, provisioning, management, and orchestration of both OpenSearch clusters and their accompanying OpenSearch Dashboards. By leveraging the Operator pattern, organizations can treat their search and analytics infrastructure as a custom resource, allowing for complex operations—such as rolling upgrades, certificate rotation, and resource scaling—to be handled by the operator rather than manual intervention.

The architecture of OpenSearch on Kubernetes is designed to be compatible with any CNCF-certified Kubernetes cluster. This compatibility ensures that whether a user is operating on a local k3s environment, a managed service like AWS EKS, or a corporate Oracle Cloud Infrastructure (OCI) deployment, the operational logic remains consistent. The integration focuses on reducing the friction associated with the initial bootstrap phase, the management of cluster quorum, and the scaling of data nodes.

Operator Installation and API Migration

The installation of the OpenSearch Operator is primarily facilitated through Helm, the standard package manager for Kubernetes. This process involves adding the specific repository and deploying the operator to the cluster, which then watches for the creation of OpenSearchCluster custom resources.

To initiate the installation, the following helm repository must be added:

helm repo add opensearch-operator https://opensearch-project.github.io/opensearch-k8s-operator/

Once the repository is available, the operator is installed using:

helm install opensearch-operator opensearch-operator/opensearch-operator

A critical architectural shift is currently occurring regarding the API group used by the operator. The system is migrating from the opensearch.opster.io API group to the opensearch.org API group. While both groups are currently supported to ensure backward compatibility, the opensearch.opster.io group is officially deprecated. Users transitioning to newer versions of the operator should consult the Migration Guide to align their manifests with the opensearch.org standard.

The impact of this migration is primarily reflected in the apiVersion field of the YAML manifests. Failure to migrate may result in deprecation warnings during deployment, although current cluster operations remain functional under the older group.

OpenSearchCluster Custom Resource Configuration

Once the operator is active, the deployment of an OpenSearch cluster is managed via the OpenSearchCluster custom resource. This allows the user to define the desired state of the cluster in a single YAML file, which the operator then reconciles.

A standard deployment configuration involves several key blocks:

• General Settings: This section defines the cluster version, the service name for internal networking, and the general port configurations.
• Security: This block manages TLS and authentication. Specifically, users can configure the operator to generate TLS certificates for both HTTP and transport layers.
• Dashboards: This section controls the deployment of OpenSearch Dashboards, including versioning and resource limits.
• Node Pools: This defines the physical and logical distribution of nodes, specifying roles and resource allocations.

The following table details the resource specifications for various components based on deployment examples:

The use of node pools allows for granular control over the cluster architecture. For instance, a user can deploy a pool of three master nodes to ensure quorum and high availability, while separately scaling data nodes to handle increased indexing load.

Advanced Operational Capabilities

The OpenSearch Kubernetes Operator provides a suite of high-level capabilities that extend beyond simple pod creation. These features are designed to maintain cluster health and ensure zero-downtime operations.

The operator supports the management of multiple node pools. This allows administrators to assign specific roles to specific nodes, such as:

• cluster_manager: Responsible for managing the cluster state.
• data: Responsible for indexing and searching data.
• ingest: Handles the preprocessing of documents.
• coordinating: Acts as a router for requests.

Further operational efficiencies include:

• Rolling version upgrades: The operator performs upgrades using quorum-safe restarts, ensuring that the cluster remains available while individual nodes are updated.
• Online volume expansion: Disk scaling can be performed without taking the node offline, allowing for seamless growth of the data layer.
• Certificate management: The operator supports TLS hot reloading, meaning certificates can be updated without requiring a full restart of the OpenSearch process.
• Multi-namespace support: This allows for the management of multiple clusters across different organizational boundaries within a single Kubernetes cluster.
• Plugin installation: Plugins, such as the repository-s3 plugin, can be installed during the bootstrap phase to ensure the cluster is fully functional upon launch.

Deployment Lifecycle and Bootstrap Process

The lifecycle of an OpenSearch cluster managed by the operator follows a specific sequence to ensure the cluster is healthy before it begins accepting traffic.

When a user applies a manifest via kubectl apply -f cluster.yaml, the operator initiates the following sequence:

1. Bootstrap Pod: The operator first creates a bootstrap pod (e.g., my-first-cluster-bootstrap-0). This pod is essential for initial master discovery, helping the other nodes in the cluster identify the leader.
2. Cluster Pods: Following the bootstrap process, the operator spins up the primary cluster pods (e.g., my-first-cluster-masters-0/1/2).
3. Dashboard Pod: Finally, a pod for the OpenSearch Dashboards instance is deployed to provide the visual interface.

To monitor this process in real-time, users can utilize the watch command:

watch -n 2 kubectl get pods

Helm Chart Deployment Alternative

While the Operator provides high-level orchestration, a direct Helm chart deployment is also available for users who prefer a more traditional Kubernetes deployment model. This method involves packaging the chart and deploying it with a values file.

The process for Helm-based deployment is as follows:

• Navigate to the chart directory: cd helm-charts/charts/opensearch
• Package the chart: helm package .
• Deploy the instance: helm install --generate-name opensearch-<VERSION>.tgz -f /path/to/values.yaml

Once deployed, the health of the cluster can be verified by listing the pods:

kubectl get pods

To verify the operational status of the API, users can execute a curl command against the service. Since OpenSearch uses HTTPS by default, the --insecure flag is often required for initial testing:

curl -XGET https://localhost:9200 -u 'admin:<custom-admin-password>' --insecure

To access the shell of a specific master node for deep troubleshooting, the following command is used:

kubectl exec -it opensearch-cluster-master-0 -- /bin/bash

Version Compatibility Matrix

The OpenSearch Kubernetes Operator is designed to remain compatible across multiple versions of OpenSearch. However, explicit testing has been performed on specific version pairings to ensure stability.

The following table outlines the compatibility between the Operator and OpenSearch versions:

It is important to note that while the operator only explicitly lists tested versions, it does not prevent the deployment of other OpenSearch versions, although stability is not guaranteed.

OpenSearch Dashboards Configuration and External Access

OpenSearch Dashboards provide the critical visualization layer for the cluster. Their deployment can be customized to fit various security and accessibility requirements.

For environments requiring custom configurations, a Kubernetes secret can be created to store the opensearch_dashboards.yml file. This ensures that configuration details are not hardcoded in the manifest:

kubectl create secret generic <opensearch-namespace>-opensearch-dashboard-config --from-file=opensearch_dashboards.yml=<path/opensearch_dashboards.yml> -n <opensearch-namespace>

To provide external access to the dashboards, an Ingress Controller is required. Nginx is recommended as a stable and configurable solution for this purpose. The integration of Nginx allows for:

• Domain-based routing: Directing traffic based on the URL used by the client.
• Path-based routing: Directing traffic based on the specific path requested.
• Traffic control: Implementing rules to manage how users access the OpenSearch environment.

This architectural choice ensures that the dashboards are both secure and scalable, preventing direct exposure of the Kubernetes service to the public internet.

Troubleshooting and Common Issues

Deploying OpenSearch on Kubernetes, particularly on managed services like AWS EKS, can encounter specific hurdles related to network connectivity and port forwarding.

A common scenario involves users attempting to verify their cluster via port-forwarding:

kubectl port-forward svc/my-cluster 9200

Errors in this stage often relate to the service name mismatch or the lack of appropriate security credentials. If the cluster is deployed using a manifest where the serviceName is defined as opensearch-mesalvo-cluster, the port-forward command must reference that specific service name.

Furthermore, when utilizing the opensearch-operator/examples/2.x/opensearch-cluster.yaml example, users must ensure that the apiVersion matches the installed operator version. A mismatch between the requested apiVersion and the available CRD (Custom Resource Definition) in the cluster will prevent the operator from recognizing the resource, leading to pods not being created.

Analysis of Deployment Strategies

The choice between using the OpenSearch Operator and direct Helm charts depends on the intended lifecycle of the cluster.

The Helm chart approach is essentially a "snapshot" deployment. It is highly effective for static environments where the cluster configuration rarely changes. However, it lacks the intelligence to handle complex operational tasks such as rolling updates or automatic certificate rotation without manual intervention or complex CI/CD pipelines.

Conversely, the Operator approach introduces a control loop. The operator constantly monitors the actual state of the cluster against the desired state defined in the OpenSearchCluster resource. If a pod fails or a version needs to be upgraded, the operator executes the necessary steps to restore the desired state. This reduces the operational burden on the DevOps team and minimizes the risk of human error during scaling events.

From a resource perspective, the operator allows for a more nuanced distribution of roles. By separating the cluster_manager from the data nodes through node pools, the cluster can be optimized for either stability (more masters) or performance (more data nodes). This flexibility is critical for production-grade deployments where data volume and query frequency can fluctuate.

Sources

1. GitHub - OpenSearch K8s Operator
2. OpenSearch Forum - Installing OpenSearch on AWS Kubernetes with K8s Operator
3. Oracle Documentation - Deploying OpenSearch and OpenSearch Dashboard
4. OpenSearch Documentation - Install OpenSearch via Helm
5. OpenSearch K8s Operator User Guide