The intersection of lightweight orchestration and enterprise automation represents a paradigm shift in how infrastructure is managed. Deploying Ansible AWX—the open-source upstream project for Ansible Automation Platform—on K3s provides a streamlined, resource-efficient environment that maintains full Kubernetes certification while removing the operational overhead typically associated with full-scale K8s distributions. This architectural choice allows administrators to leverage the AWX Operator for lifecycle management, utilizing a declarative approach to deploy a complex application stack consisting of a PostgreSQL database, a task engine, and a web-based user interface. By utilizing K3s, which incorporates an integrated Traefik ingress controller and a local-path provisioner for storage, the deployment process is significantly simplified, transforming what was once a grueling manual installation into a repeatable, code-driven operation.
The Foundation: K3s Infrastructure and Host OS
The viability of an AWX deployment begins with the selection of the underlying operating system and the Kubernetes distribution. A primary recommendation for stability and enterprise readiness is the use of Rocky Linux 8.9 (Green Obsidian). Rocky Linux is engineered as an open-source, enterprise-ready operating system that rebuilds sources directly from Red Hat Enterprise Linux (RHEL). This ensures a high degree of binary compatibility with RHEL, providing a stable foundation with a predicted 10-year support lifecycle, which is critical for production automation controllers that must remain operational for years without disruptive OS migrations.
K3s is selected as the orchestration layer because it is a lightweight, certified Kubernetes distribution. Unlike standard K8s, which requires complex bootstrapping and multiple components, K3s is packaged as a single binary. This reduction in complexity does not compromise the Kubernetes API, as K3s remains fully compliant with the main way Kubernetes operates.
The installation of K3s on a Rocky Linux host is executed via a streamlined shell command:
curl -sfL https://get.k3s.io | sh -
Once the installation is complete, the operational status of the cluster can be verified by querying the nodes and pods to ensure the control plane is healthy:
kubectl get nodes
kubectl get pods
Implementing the AWX Operator via Kustomize
The modern method for deploying AWX is through the AWX Operator, which manages the lifecycle of the AWX instance. To implement this, a directory structure must be established to house the configuration manifests.
mkdir k3s
cd k3s
touch kustomization.yaml
The kustomization.yaml file acts as the orchestrator for the deployment, defining which resources to pull and how to modify them. The following configuration demonstrates a deployment using version 2.5.2 of the operator:
yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- github.com/ansible/awx-operator/config/default?ref=2.5.2
- awx-demo.yml
images:
- name: quay.io/ansible/awx-operator
newTag: 2.5.2
namespace: awx
The use of Kustomize allows the administrator to specify a custom namespace (awx) and override image tags to ensure version consistency. The manifests are then applied to the cluster using the following command:
kubectl apply -k .
The output of this command confirms the creation of several critical components:
- The awx namespace.
- Custom Resource Definitions (CRDs) for awxbackups, awxrestores, and awxs, which allow Kubernetes to recognize AWX-specific objects.
- Service accounts and RBAC roles (e.g., awx-operator-awx-manager-role, awx-operator-proxy-role) that grant the operator the necessary permissions to manage pods and services.
- A ConfigMap for the operator manager.
Configuring the AWX Instance (The CRD Layer)
After the operator is running, a Custom Resource (CR) must be created to tell the operator to actually deploy an AWX instance. This is done via a file named awx-demo.yml.
```yaml
apiVersion: awx.ansible.com/v1beta1 kind: AWX metadata: name: awx-demo spec: servicetype: clusterip ingresstype: ingress ```
In this configuration, the service_type is set to clusterip, meaning the service is only accessible within the cluster. To expose the web interface to external users, the ingress_type is set to ingress. This is a critical decision point: K3s includes Traefik by default, which means the ingress will work "out of the box" without additional configuration. In contrast, other distributions like minikube require the ingress addon to be manually enabled via minikube.exe addons enable ingress.
The resource is applied using:
kubectl apply -f awx.demo.yml
Following this command, the operator initiates the creation of the database pod and the AWX application pod. This process requires patience as the operator configures the database and prepares the application environment. To streamline management, users can set their current context to the awx namespace to avoid repeating -n awx in every command:
kubectl config set-context --current --namespace=awx
Storage and Persistence Management
AWX relies heavily on persistent storage for its database and project files. K3s provides a default local-path storage class, which maps Kubernetes Persistent Volume Claims (PVCs) to local directories on the host machine.
A typical PVC for AWX projects might appear as follows:
| Component | Status | Volume | Capacity | Access Mode | Storage Class |
|---|---|---|---|---|---|
| awx-projects-claim | Bound | pvc-ba714636... | 8Gi | RWO | local-path |
The physical location of these files on the host system is critical for manual backups or direct file manipulation. In K3s, the project directory is located at:
/var/lib/rancher/k3s/storage/<volume>_<namespace>_<pvc name>/
For example, a specific volume would be found at /var/lib/rancher/k3s/storage/pvc-ba714636-15d9-4789-bcf8-3cb530d809b7_awx_awx-projects-claim/.
It is important to note that the local-path provisioner is not a CSI (Container Storage Interface) driver and does not support native snapshotting. For enterprise-grade backups, users are encouraged to use Kasten generic backups or implement a dedicated CSI-hostpath driver if snapshotting capabilities are required.
Networking, TLS, and Host Resolution
To access the AWX interface via a browser, the system must resolve the host name. This is often achieved by defining an AWX_HOST variable and generating self-signed certificates for secure communication.
export AWX_HOST="awx-k3s.demo.local"
The generation of the TLS certificate and key is performed using OpenSSL:
openssl req -x509 -nodes -days 3650 -newkey rsa:2048 -out ./awx/tls.crt -keyout ./awx/tls.key -subj "/CN=${AWX_HOST}/O=${AWX_HOST}" -addext "subjectAltName = DNS:${AWX_HOST}"
Once the certificates are generated and the Kustomize manifests are applied (kubectl apply -k awx), the ingress controller (Traefik) routes external traffic to the AWX service. Users can then navigate to the specified host in a web browser to access the AWX landing page.
Challenges of Offline (Air-Gapped) Installations
Deploying AWX in an offline environment introduces significant complexities, as the Kubelet and the AWX Operator are designed to pull container images from remote registries (such as quay.io).
A common failure mode occurs when a system is installed with internet access and then moved to an offline environment. The system will continuously attempt to connect to the internet to pull updated images or configurations, leading to job failures.
The most critical issue relates to Execution Environments (EE). EE images are used to run Ansible playbooks. Even if the images are initially present, they can be deleted by the Kubelet's garbage collection process if disk pressure increases. If the cached images are deleted in an offline environment, the job will fail to start because it cannot pull the image from the internet.
To resolve this, the following strategy is recommended:
- Store all required EE images in a private container registry within the offline network.
- Configure the EE to point to this private registry.
- Set the pull policy to Missing or Always to ensure the system looks at the internal registry rather than the default external ones.
Summary of Component Interaction
The following table summarizes the relationship between the K3s components and the AWX deployment:
| K3s Component | AWX Dependency | Role/Impact |
|---|---|---|
| Traefik | Ingress | Enables external browser access to the AWX UI |
| Local-Path Provisioner | PVC | Stores PostgreSQL data and project files |
| Kubelet | Pod Management | Handles image pulling and garbage collection |
| API Server | AWX Operator | Process the AWX Custom Resource and manage pods |
| Rocky Linux 8.9 | Host OS | Provides the stable, RHEL-compatible kernel and binaries |
Conclusion
The deployment of Ansible AWX on K3s is a sophisticated exercise in utilizing the Kubernetes operator pattern to simplify the management of a complex application. By leveraging Rocky Linux 8.9 for stability and K3s for lightweight orchestration, administrators can create a powerful automation hub that is easily reproducible. The integration of Kustomize allows for precise version control of the AWX Operator, while the use of Traefik ensures a seamless path from the network to the application.
However, the transition to offline environments remains a primary technical hurdle. The reliance on external registries for Execution Environments means that a robust internal registry strategy is not optional but mandatory for air-gapped stability. Furthermore, the use of local-path storage requires a conscious decision regarding backup strategies, as the lack of CSI-native snapshots necessitates third-party tools like Kasten for data integrity. Ultimately, the success of this architecture lies in the understanding that while the "out of the box" experience is streamlined, the operational requirements for persistence, security (TLS), and image availability require a deep dive into the underlying Kubernetes mechanics.