The deployment of Ansible AWX upon a K3s infrastructure represents a strategic convergence of enterprise-grade automation and streamlined container orchestration. Ansible AWX, the open-source upstream project for Ansible Automation Controller, provides a sophisticated web-based interface and REST API for managing Ansible playbooks, inventories, and project synchronization. When paired with K3s—a certified, lightweight Kubernetes distribution developed by Rancher—organizations can achieve a highly efficient footprint that reduces the overhead typically associated with standard Kubernetes (K8s) while maintaining full API compatibility. This architectural choice allows for the rapid deployment of automation hubs on hardware ranging from cloud-based virtual machines to edge computing devices, provided the minimum resource thresholds are met. The integration relies heavily on the AWX Operator, a Kubernetes controller that manages the lifecycle of the AWX instance, including the deployment of the PostgreSQL database, the web interface, and the task manager. By leveraging the Operator pattern, the complex process of configuring persistent storage and managing containerized dependencies is abstracted into a declarative set of Custom Resource Definitions (CRDs), ensuring that the state of the automation hub is consistently maintained across the cluster.
Core System Requirements and Environment Preparation
The stability of an AWX deployment is fundamentally dependent on the underlying hardware and operating system. Failure to meet the minimum resource specifications typically results in catastrophic deployment failures, as the AWX Operator and the subsequent pods (web, task, and database) require significant memory and compute for successful initialization and operation.
Hardware Resource Specifications
The baseline requirements for a successful deployment are as follows:
- CPU: Minimum of 4 CPU cores.
- RAM: Minimum of 4GB of memory.
Technical Layer: The 4GB RAM threshold is critical because the AWX deployment consists of multiple resource-intensive pods. The PostgreSQL database and the AWX web/task containers utilize a significant amount of memory for caching and executing Python-based automation tasks. If the system falls below these specifications, the Kubernetes scheduler may trigger Out-Of-Memory (OOM) kills, leading to a crash loop (CrashLoopBackOff) for the pods.
Impact Layer: For the end-user, ignoring these specifications leads to a non-functional installation where the web interface fails to load or the task manager cannot execute playbooks. This necessitates a hardware upgrade or a virtual machine reallocation before the installation can proceed.
Contextual Layer: These hardware requirements must be considered alongside the choice of the host operating system to ensure that the kernel and system libraries are optimized for K3s and container runtimes.
Operating System Selection
While various Linux distributions are supported, specific enterprise-grade options are recommended for stability:
- Oracle Linux 8: Preferred for its stability, free availability, and robust support for Docker and Kubernetes ecosystems.
- Rocky Linux 8.9 (Green Obsidian): An open-source enterprise-ready OS rebuilt from RHEL sources, offering a 10-year support lifecycle and high stability.
Technical Layer: Both Oracle Linux 8 and Rocky Linux 8 provide a binary-compatible environment with Red Hat Enterprise Linux (RHEL). This ensures that the systemd configurations, networking stacks, and security modules (like SELinux) are predictable and compatible with the K3s installation scripts.
Impact Layer: Using these distributions ensures that the administrator has a stable base that won't suffer from unexpected regressions during kernel updates, which is vital for a system managing critical infrastructure automation.
Contextual Layer: The choice of OS directly influences the initial setup steps, such as the installation of the K3s binary and the configuration of user permissions for the kubectl tool.
K3s Cluster Installation and Configuration
K3s is a lightweight distribution of Kubernetes that removes unnecessary legacy code and ships as a single binary, making it an ideal host for the AWX Operator.
Initial K3s Deployment
The installation of K3s is executed via a streamlined bash script that automates the pulling of the binary and the configuration of the systemd service.
Execution command:
curl -sfL https://get.k3s.io | sh -
Technical Layer: The curl command retrieves the installation script from the official K3s repository and pipes it directly into the shell. This process installs the K3s server, configures the local storage path for data, and initializes the cluster.
Impact Layer: This allows the user to move from a clean OS install to a functional Kubernetes cluster in minutes, removing the need for complex manual configuration of the kubelet and kube-proxy.
Contextual Layer: Once the installation is complete, the administrator must verify the cluster health using kubectl get nodes and kubectl get pods to ensure the control plane is operational before proceeding to the AWX Operator installation.
Administrative Permission Management
By default, the K3s configuration file is owned by the root user, which restricts the ability of standard users to interact with the cluster.
Command to change permissions:
sudo chown $USER:$USER /etc/rancher/k3s/k3s.yaml
Technical Layer: The file /etc/rancher/k3s/k3s.yaml contains the cluster's kubeconfig. By changing the ownership to the current user, the administrator can run kubectl commands without needing to prefix every command with sudo.
Impact Layer: This improves the developer experience and prevents potential security issues associated with running the entire shell as root during the deployment of AWX manifests.
Contextual Layer: This step is a prerequisite for using Kustomize or applying YAML files, as these tools rely on the accessibility of the kubeconfig file.
Deploying the AWX Operator via Kustomize
The AWX Operator is the engine that manages the AWX instance. It is typically deployed using Kustomize, which allows for the customization of Kubernetes manifests without modifying the original source code.
Kustomize Installation and Setup
Kustomize must be installed on the host machine to build the operator manifests.
Installation steps:
curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash
sudo mv kustomize /usr/local/bin/
Technical Layer: Kustomize is a template-free manifest customization tool. It allows the administrator to specify image tags and namespaces that differ from the defaults provided by the Ansible team.
Impact Layer: This provides the flexibility to use specific versions of the AWX Operator, which is essential for maintaining compatibility with specific versions of the AWX application.
Contextual Layer: Kustomize serves as the bridge between the generic AWX Operator source code on GitHub and the specific requirements of the local K3s cluster.
Configuring the Kustomization Manifest
A kustomization.yaml file is created to define the resources and image tags.
Directory setup:
mkdir k3s
cd k3s
touch kustomization.yaml
Example kustomization.yaml content:
yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- github.com/ansible/awx-operator/config/default?ref=2.5.2
- awx-demo.yml
images:
- name: quay.io/ansible/awx-operator
newTag: 2.5.2
namespace: awx
Technical Layer: The resources section points to the remote AWX Operator configuration. The images section overrides the default image tag with a specific version (e.g., 2.5.2 or 2.2.1), ensuring the cluster pulls the exact version of the operator desired. The namespace: awx directive ensures all operator components are isolated within a dedicated namespace.
Impact Layer: This declarative approach means the installation is reproducible. If the cluster needs to be rebuilt, the same kustomization.yaml file can be used to recreate the exact environment.
Contextual Layer: After creating this file, the operator is installed using the command kubectl apply -k ., which triggers the creation of several ClusterRoles, ServiceAccounts, and CustomResourceDefinitions (CRDs).
Operator Installation Execution
The installation is finalized by applying the Kustomize build to the cluster.
Command:
kubectl apply -k .
Technical Layer: This command instructs Kubernetes to process the kustomization.yaml and create the necessary objects. The output will confirm the creation of awxbackups.awx.ansible.com, awxrestores.awx.ansible.com, and awxs.awx.ansible.com CRDs.
Impact Layer: The creation of these CRDs allows the user to define an "AWX" object in YAML, which the operator then watches and implements by spawning the actual AWX pods.
Contextual Layer: Once the operator is running, the administrator can proceed to deploy the specific AWX instance using a deployment YAML file.
AWX Instance Deployment and Persistence
With the operator active, the actual AWX application must be deployed. This involves defining the AWX instance and ensuring that data is stored on persistent volumes to prevent data loss during pod restarts.
Persistent Volume Claim (PVC) Setup
To ensure that project data and database information persist, a PersistentVolumeClaim must be defined.
File: public-static-pvc.yaml
```yaml
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: public-static-data-pvc spec: accessModes: - ReadWriteOnce storageClassName: local-path resources: requests: storage: 20Gi ```
Technical Layer: This PVC requests 20Gi of storage from the local-path storage class, which is the default for K3s. The ReadWriteOnce access mode ensures that the volume can be mounted by a single node at a time, which is sufficient for a single-node K3s setup.
Impact Layer: Without this PVC, any data stored in /var/lib/projects would be ephemeral, meaning all uploaded project files would be deleted if the pod is rescheduled.
Contextual Layer: This PVC is referenced in the awx-instance-deployment.yml file to mount the storage into the web container.
AWX Instance Configuration
The AWX instance deployment file defines the behavior and service type of the application.
File: awx-instance-deployment.yml
```yaml
apiVersion: awx.ansible.com/v1beta1 kind: AWX metadata: name: awx spec: servicetype: nodeport projectspersistence: true projectsstorageaccessmode: ReadWriteOnce webextravolumemounts: | - name: static-data mountPath: /var/lib/projects extra_volumes: | - name: static-data persistentVolumeClaim: claimName: public-static-data-pvc ```
Technical Layer: The service_type: nodeport allows the AWX web interface to be accessible from outside the cluster using the node's IP address and a specific port. The projects_persistence: true setting tells the operator to enable storage for AWX projects, and the web_extra_volume_mounts connects the previously defined PVC to the container's file system.
Impact Layer: This configuration transforms AWX from a transient container into a durable application capable of storing automation playbooks and maintaining a database of inventories.
Contextual Layer: These files are applied using kubectl apply -f public-static-pvc.yaml -n awx and kubectl apply -f awx-instance-deployment.yml -n awx.
Monitoring the Deployment Process
The AWX Operator manages the creation of several pods. The administrator should monitor the progress to ensure the database and web components start correctly.
Monitoring command:
kubectl get pods -l "app.kubernetes.io/managed-by=awx-operator" -n awx -w
Technical Layer: This command filters pods by the label assigned by the operator. The expected pods include awx-postgres, awx-task, and awx-web. The -w flag enables "watch" mode, providing real-time updates on the pod status.
Impact Layer: This allows the user to identify if a pod is stuck in ContainerCreating or ImagePullBackOff, which often indicates networking issues or insufficient resources.
Contextual Layer: Once the pods are in the Running state, the user can identify the assigned NodePort to access the web interface.
Special Considerations for Offline Environments
Deploying AWX in an air-gapped or offline environment introduces complexities regarding image availability and configuration pulls.
Image Management and Private Registries
In an offline environment, K3s cannot pull images from quay.io or github.com. The system will continuously attempt to connect to the internet, which can lead to timeouts and deployment failures.
Technical Layer: To resolve this, all required container images must be manually mirrored to a private container registry located within the offline network. This includes the AWX Operator image, the AWX web image, the task image, and the PostgreSQL image.
Impact Layer: If images are not hosted locally, the Kubelet will fail to start the pods, and the AWX instance will never reach a Running state.
Contextual Layer: This is particularly critical for Execution Environments (EE), which are the containerized environments where Ansible playbooks actually run.
Execution Environment (EE) Cache and Garbage Collection
A significant risk in offline setups is the Kubelet's garbage collection mechanism.
Technical Layer: Kubelet may delete cached images to save disk space. If an EE image is deleted from the local cache and the cluster is offline, the job will fail because it cannot pull the image from the internet.
Impact Layer: To prevent this, the administrator must configure the EE to point to a private registry with the pull policy set to Missing or Always. This ensures that the system knows exactly where to find the image without relying on the volatile local cache.
Contextual Layer: This requires modifying the AWX instance configuration to specify the internal registry URL for all execution environments.
Summary of Deployment Specifications
The following table summarizes the critical components and requirements for an AWX on K3s deployment.
| Component | Specification/Value | Purpose |
|---|---|---|
| Minimum CPU | 4 Cores | Prevent pod crash loops |
| Minimum RAM | 4 GB | Ensure stable execution of AWX components |
| OS Options | Oracle Linux 8 / Rocky Linux 8 | Enterprise stability and RHEL compatibility |
| K3s Distribution | Lightweight Kubernetes | Reduced overhead, API compliant |
| Operator Version | 2.5.2 / 2.2.1 | Lifecycle management of AWX |
| Storage Class | local-path | Default K3s persistent storage |
| PVC Size | 20 Gi | Storage for projects and static data |
| Service Type | NodePort | External access to the AWX web UI |
Conclusion: Analytical Review of the Architecture
The deployment of Ansible AWX on K3s creates a highly efficient automation hub that balances the power of Kubernetes with the simplicity of a lightweight footprint. By utilizing the Operator pattern, the complexity of deploying a multi-component application—consisting of a database, a web server, and a task engine—is reduced to a few declarative YAML files. The use of Kustomize provides the necessary abstraction to manage versions and namespaces without mutating the upstream source code.
However, the architecture reveals a critical dependency on resource availability. The failure of the system to meet the 4GB RAM threshold is a common point of failure, emphasizing that while K3s is "lightweight," the applications it hosts (like AWX) still possess enterprise-level resource demands. Furthermore, the transition from an online to an offline environment is not a simple matter of disconnecting the network; it requires a strategic shift toward private registry management to bypass the Kubelet's reliance on external image repositories and the risks associated with image garbage collection.
Ultimately, this setup provides a scalable foundation. Starting with a single-node K3s cluster on Rocky Linux or Oracle Linux 8 allows for an easy transition to a multi-node cluster in the future, as the AWX Operator handles the orchestration of the pods regardless of the cluster size, provided the persistent storage is correctly configured via PVCs.