The deployment of Ansible AWX upon a K3s Kubernetes cluster represents a sophisticated convergence of enterprise automation and lightweight container orchestration. AWX, the open-source upstream project for Ansible Automation Platform, provides a critical layer of role-based access control, inventory management, and job scheduling. When paired with K3s—a certified, lightweight Kubernetes distribution engineered by Rancher—the result is a highly efficient, low-overhead management plane capable of orchestrating complex infrastructure tasks. This integration is particularly potent for engineers who require the full power of Kubernetes without the resource intensity of a standard K8s distribution, making it ideal for edge computing, single-node management servers, or development environments.
The operational synergy between AWX and K3s is managed primarily through the AWX Operator. The operator pattern in Kubernetes is a method of extending the Kubernetes API to manage complex applications. Instead of manually configuring every pod, service, and ingress, the operator handles the lifecycle of the AWX instance, including the deployment of the PostgreSQL database, the task engine, and the web user interface. By utilizing K3s as the underlying substrate, the infrastructure footprint is significantly reduced, allowing for rapid deployment on hardware that would typically be insufficient for a full-scale production Kubernetes cluster.
Hardware and Operating System Prerequisites
Successful deployment of the AWX stack on K3s requires a specific baseline of computational resources. Failure to meet these minimums typically results in deployment failures, pod crashes, or "Pending" states due to insufficient CPU or Memory requests.
Resource Requirements
The minimum hardware specifications required for a stable deployment are as follows:
| Resource | Minimum Requirement | Technical Justification |
|---|---|---|
| CPU Cores | 4 Cores | Required to handle the simultaneous demands of the AWX Operator, Postgres database, and the Task/Web pods. |
| RAM | 4 GB | Necessary to prevent Out-Of-Memory (OOM) kills during the initial database migration and operator startup. |
| OS | Linux | Specifically recommended distributions include Rocky Linux 8.9 or Oracle Linux 8. |
Operating System Selection and Rationale
The choice of operating system significantly impacts the stability of the K3s runtime.
Rocky Linux 8.9 (Green Obsidian) is highly recommended because it is an open-source, enterprise-ready operating system rebuilt directly from Red Hat Enterprise Linux (RHEL) sources. This provides a 10-year support lifecycle and high stability, which is critical for the long-term reliability of an automation hub. Similarly, Oracle Linux 8 is favored for its stability and robust support for both Docker and Kubernetes ecosystems. Utilizing a minimal installation of these distributions ensures that the kernel is not bloated with unnecessary packages, maximizing the available RAM for the K3s cluster.
K3s Cluster Initialization and Configuration
K3s is a streamlined version of Kubernetes that removes legacy, alpha, and non-essential code to reduce the binary size and memory footprint. It is fully certified as Kubernetes, meaning it adheres to all standard K8s APIs.
Installation Process
The installation of K3s is performed using a streamlined shell script that automates the process of downloading the binary and configuring the systemd service.
The command to initiate the installation is:
curl -sfL https://get.k3s.io | sh -
Following the installation, the operator must verify that the node is operational and joined to the cluster. This is achieved by querying the cluster state:
kubectl get nodes
kubectl get pods
Permission Management and Environment Setup
By default, the K3s configuration file is restricted to the root user for security reasons. To allow a non-root user to manage the cluster, ownership of the configuration file must be modified.
The command to change permissions is:
sudo chown $USER:$USER /etc/rancher/k3s/k3s.yaml
This action allows the current user to execute kubectl commands without requiring sudo for every interaction, which is essential for integrating with CI/CD pipelines or automated scripts.
Deploying the AWX Operator via Kustomize
The AWX Operator is the engine that manages the AWX deployment. Using Kustomize allows engineers to customize the operator's deployment without modifying the original manifest files.
Kustomize Installation
Kustomize must be installed on the host system to build the manifests. The installation follows these steps:
- Download the installation script:
curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash - Move the binary to the system path:
sudo mv kustomize /usr/local/bin/
Manifest Configuration
The operator is deployed by creating a kustomization.yaml file. This file defines the resources to be pulled and the specific versions (tags) of the images to be used.
For a deployment utilizing version 2.5.2, the kustomization.yaml is structured as follows:
yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- github.com/ansible/awx-operator/config/default?ref=2.5.2
- awx-demo.yml
images:
- name: quay.io/ansible/awx-operator
newTag: 2.5.2
namespace: awx
For users utilizing version 2.2.1, the resource reference is adjusted to ref=2.2.1 and the newTag is set to 2.2.1.
Operator Execution
The manifests are applied to the cluster using the kubectl apply -k . command. This process triggers the creation of several critical Kubernetes objects:
- Namespace:
awxis created to isolate the AWX resources. - Custom Resource Definitions (CRDs): These include
awxbackups.awx.ansible.com,awxrestores.awx.ansible.com, andawxs.awx.ansible.com. These definitions tell Kubernetes how to handle "AWX" as a native object. - RBAC Roles:
awx-operator-controller-manager,awx-operator-awx-manager-role, and others are created to grant the operator the necessary permissions to manage pods and services. - ConfigMaps: The
awx-operator-awx-manager-configis established to store operational settings.
AWX Instance Deployment and Persistence
Once the operator is running, an actual AWX instance must be defined. This involves creating a persistent volume for data and a deployment manifest for the instance itself.
Persistent Volume Claim (PVC)
To ensure that AWX data survives pod restarts or cluster reboots, a PersistentVolumeClaim is required. This is defined in a file named public-static-pvc.yaml.
```yaml
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: public-static-data-pvc spec: accessModes: - ReadWriteOnce storageClassName: local-path resources: requests: storage: 20Gi ```
The use of storageClassName: local-path indicates that K3s will use its default local path provisioner to allocate 20GB of disk space on the node.
Instance Deployment Manifest
The AWX instance is defined in awx-instance-deployment.yml. This manifest tells the operator how to configure the AWX application.
```yaml
apiVersion: awx.ansible.com/v1beta1 kind: AWX metadata: name: awx spec: servicetype: nodeport projectspersistence: true projectsstorageaccessmode: ReadWriteOnce webextravolumemounts: | - name: static-data mountPath: /var/lib/projects extra_volumes: | - name: static-data persistentVolumeClaim: claimName: public-static-data-pvc ```
Key technical details in this manifest include:
- service_type: nodeport: This exposes the AWX web interface on a specific port of the K3s node IP.
- projects_persistence: true: Ensures that project data is stored on the persistent volume.
- web_extra_volume_mounts: Maps the static-data volume to /var/lib/projects within the container, allowing for the storage of playbooks and project files.
Execution and Monitoring
The manifests are applied sequentially:
kubectl apply -f public-static-pvc.yaml -n awx
kubectl apply -f awx-instance-deployment.yml -n awx
The deployment progress can be monitored using a label selector to filter for pods managed by the operator:
kubectl get pods -l "app.kubernetes.io/managed-by=awx-operator" -n awx -w
A successful deployment will result in the following pods reaching a "Running" state:
- awx-postgres-13-0: The database backend.
- awx-task-XXXXX: The background task engine.
- awx-web-XXXXX: The web user interface.
Offline Installation and Air-Gapped Strategies
Installing AWX on K3s in an offline environment introduces significant challenges, as the Kubelet and the Operator normally attempt to pull images from external registries like quay.io or docker.io.
The "Bridge" Method
One method for achieving an offline installation is to perform a full installation on a system with internet access using the kurokobo/awx-on-k3s implementation, and then physically or logically moving that system into the offline environment.
However, this method faces a critical failure point: the Kubelet's garbage collection. Kubelet may delete cached images to free up space. If the cached images for the AWX operator or the Execution Environments (EE) are deleted, the pods will fail to start because they cannot reach the internet to re-pull the images.
Private Registry Implementation
To ensure a robust offline deployment, a private container registry must be established within the air-gapped environment.
- Image Migration: All required images (AWX Operator, Postgres, AWX Web, and AWX Task) must be pulled on an internet-connected machine, saved as tarballs using
docker save, moved to the offline environment, and pushed to the private registry. - Execution Environment (EE) Configuration: EE images are used to run Ansible playbooks. In an offline setup, the AWX instance must be configured to point to the private registry.
- Pull Policy: The image pull policy should be set to
MissingorAlways. If set toAlways, Kubelet will check the private registry for a newer version; if set toMissing, it will only pull if the image is not present locally.
Technical Troubleshooting and Operational Analysis
Common Failure Modes
The most frequent cause of deployment failure is resource exhaustion. If the 4 CPU/4GB RAM threshold is not met, the awx-postgres pod may enter a CrashLoopBackOff state due to timeout during database initialization.
Another common issue is permission errors on the K3s config file. If the user receives a "forbidden" error when running kubectl, the fix is to ensure the k3s.yaml ownership is correctly set to the active user.
Accessing the AWX Interface
Once the pods are running, the user must identify the NodePort assigned to the service to access the web UI. This is done by inspecting the services in the awx namespace:
kubectl get svc -n awx
The output provides the port mapping (e.g., 80:3xxxx), where 3xxxx is the port to be used in the browser along with the server's IP address.
Conclusion
The deployment of Ansible AWX on K3s is a powerful strategy for consolidating automation management. By leveraging a lightweight Kubernetes distribution, organizations can deploy a full-featured automation controller without the overhead of a standard cluster. The use of the AWX Operator streamlines the lifecycle management of the application, while the integration of Persistent Volume Claims ensures data durability. In air-gapped environments, the shift from relying on local image caches to a dedicated private registry is mandatory to prevent catastrophic failures caused by Kubelet garbage collection. This architecture provides a scalable, maintainable, and highly efficient foundation for enterprise-grade Ansible automation.