Architectural Orchestration of AWX on Kubernetes via the AWX Operator

The deployment of AWX, the open-source upstream project for Ansible Tower, represents a paradigm shift in how organizations manage automation. By providing a sophisticated web-based user interface, a robust REST API, and granular role-based access control (RBAC), AWX transforms raw Ansible playbooks into an enterprise-grade automation platform. While AWX can be installed in various environments, the industry-standard approach is deployment via Kubernetes. This methodology leverages the AWX Operator, a specialized controller that manages the lifecycle of AWX instances, ensuring that the complex dependencies of the application—such as the PostgreSQL database, the web interface, and the task engine—are provisioned and maintained in a consistent, declarative state.

The use of an operator pattern is critical because AWX is not a simple stateless application. It requires a coordinated dance between the database schema migrations, the deployment of the web pod, and the initialization of the task worker. By utilizing the AWX Operator, administrators move away from manual installation scripts toward a "GitOps" friendly model where the desired state of the AWX cluster is defined in a YAML manifest, and the operator works tirelessly to reconcile the actual state of the cluster with that definition.

Infrastructure Prerequisites and Resource Planning

Before initiating the deployment of AWX, the underlying Kubernetes infrastructure must be validated to ensure stability and performance. The flexibility of the AWX Operator allows it to run on a variety of distributions, including lightweight options like k3s or minikube, as well as enterprise cloud offerings such as Amazon Elastic Kubernetes Service (EKS), Google Kubernetes Engine (GKE), and Azure Kubernetes Service (AKS).

The hardware requirements are non-negotiable for a stable production or development environment. At a minimum, the cluster must provide 4GB of free memory and 2 CPU cores. This resource allocation is necessary because AWX runs several heavy components: the operator itself, the PostgreSQL database instance, the web frontend, and the task manager. Insufficient resources often lead to "CrashLoopBackOff" errors or severe latency in the web interface.

A critical technical requirement is the presence of a StorageClass that supports dynamic provisioning. Because AWX relies on PostgreSQL for all its configuration and job history data, the cluster must be able to automatically provision Persistent Volumes (PVs). If a dynamic provisioner is not available, administrators must manually define Persistent Volumes and Persistent Volume Claims (PVCs), particularly when using local storage paths.

To verify the health and readiness of the cluster, the following diagnostic commands must be executed:

bash kubectl cluster-info

This command confirms that the kubectl client is properly authenticated and communicating with the Kubernetes API server. To analyze resource availability and ensure the 4GB/2CPU requirement is met, the following command is used:

bash kubectl top nodes

Finally, to ensure the storage layer is prepared for the PostgreSQL deployment, the existing storage classes are listed via:

bash kubectl get storageclass

The AWX Operator: Lifecycle Management and Deployment

The AWX Operator is the central intelligence of the installation. Originally developed in 2019 by Jeff Geerling and now maintained by the official Ansible Team, the operator is designed to be deployed within a Kubernetes cluster to manage the entire lifecycle of an AWX instance within a specific namespace.

The operator's primary role is to watch for "Custom Resources" (CRs) of the kind AWX. When a user applies a YAML file defining an AWX instance, the operator detects this request and begins the sequential process of deploying the necessary pods. This removes the need for the user to manually manage the order of operations for the database and the application.

For those utilizing Helm for package management, the AWX Operator can be deployed using the official Helm charts. The documentation for these charts is hosted at https://ansible-community.github.io/awx-operator-helm/, while the general operator documentation is found at https://ansible.readthedocs.io/projects/awx-operator/.

The deployment process typically begins with the creation of a dedicated namespace to ensure logical isolation of AWX components from other cluster workloads:

bash kubectl create namespace awx

Advanced Storage Configuration and Local Volume Management

In many environments, especially on-premises or in edge computing scenarios using k3s, dynamic cloud storage is unavailable. In these cases, a local-storage configuration must be implemented. This requires a three-tier definition: the StorageClass, the Persistent Volume (PV), and the Persistent Volume Claim (PVC).

The StorageClass must be configured with the kubernetes.io/no-provisioner provisioner and a volumeBindingMode set to WaitForFirstConsumer. This ensures that the volume is only bound when a pod is scheduled to a specific node.

The following technical specifications are required for a local storage implementation on a node named ubuntu-4gb-nbg1-2 with a data path at /mnt/storage:

yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: local-storage namespace: awx provisioner: kubernetes.io/no-provisioner volumeBindingMode: WaitForFirstConsumer

To map the physical disk to the cluster, a Persistent Volume must be defined:

yaml apiVersion: v1 kind: PersistentVolume metadata: name: postgres-15-pv namespace: awx spec: capacity: storage: 10Gi accessModes: - ReadWriteOnce local: path: /mnt/storage storageClassName: local-storage nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - ubuntu-4gb-nbg1-2

The corresponding Persistent Volume Claim (PVC) allows the PostgreSQL pod to request this specific storage:

yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: postgres-15-awx-postgres-15-0 namespace: awx spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: local-storage

This granular control over storage is essential for data persistence. Without it, any restart of the PostgreSQL pod would result in the complete loss of all AWX configurations, users, and job histories.

Automated Deployment via Ansible and Helm

For organizations seeking to automate the deployment of the AWX Operator itself, an Ansible playbook utilizing the kubernetes.core collection is the preferred method. This approach allows the infrastructure to be treated as code, ensuring repeatability across multiple environments.

The automation flow involves several distinct tasks:

• Preparation of the host: Ensuring the postgres user exists and the /mnt/storage directory is created with 0755 permissions and correct ownership.
• Helm Repository Management: Adding the official repository using helm repo add awx-operator https://ansible-community.github.io/awx-operator-helm/ and updating the local cache.
• Namespace and StorageClass Creation: Using the kubernetes.core.k8s module to ensure the awx namespace and the local-storage class are present.

The custom resource for the AWX instance is then applied. A typical specification for a NodePort deployment looks like this:

yaml apiVersion: awx.ansible.com/v1beta1 kind: AWX metadata: name: awx namespace: awx spec: service_type: nodeport postgres_storage_class: local-storage

Post-Installation Validation and Troubleshooting

Once the AWX Custom Resource is applied, the operator begins the deployment. The user must monitor the pods to ensure they reach a Running state. The expected pods are:

• awx-operator-controller-manager-xxx: The brain that manages the AWX instance.
• awx-postgres-13-0: The database engine.
• awx-web-xxx: The user interface and API server.
• awx-task-xxx: The engine that executes the Ansible playbooks.

To check the status of these pods:

bash kubectl -n awx get pods

If the deployment stalls, troubleshooting is required. Migration issues are the most common cause of failure, as the AWX web pod often waits for the database to be fully migrated. This can be diagnosed by checking the logs of the task pod:

bash kubectl -n awx logs deployment/awx-task -c awx-task --tail=100

To verify that the PostgreSQL database is actually reachable and responding to queries, an administrator can execute a command directly inside the database pod:

bash kubectl -n awx exec -it awx-postgres-13-0 -- psql -U awx -d awx -c "SELECT 1"

Accessing the AWX Interface and Production Networking

After the pods are running, the system generates an initial administrator password. This password is stored as a Kubernetes Secret and must be decoded from base64 to be used:

bash kubectl -n awx get secret awx-admin-password -o jsonpath='{.data.password}' | base64 --decode

The method of accessing the web interface depends on the service_type defined in the AWX specification.

For those using nodeport (common in minikube or local test environments), the port can be found via:

bash kubectl -n awx get svc awx-service

In a minikube environment, the specific URL can be generated using:

bash minikube service awx-service -n awx --url

For cloud deployments using a LoadBalancer, the administrator must wait for the cloud provider to assign an external IP address, which is also visible via the get svc command.

For production environments, using a NodePort or LoadBalancer IP is insufficient. A proper Ingress resource should be implemented to handle DNS, TLS termination, and routing. The following Ingress configuration is recommended for Nginx controllers:

yaml apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: awx-ingress namespace: awx annotations: nginx.ingress.kubernetes.io/proxy-body-size: "0" nginx.ingress.kubernetes.io/proxy-read-timeout: "600" nginx.ingress.kubernetes.io/proxy-send-timeout: "600" spec: ingressClassName: nginx rules: - host: awx.example.com http: paths: - path: / pathType: Prefix backend: service: name: awx-service port: number: 80 tls: - hosts: - awx.example.com secretName: awx-tls

The annotations for proxy-body-size and proxy-read-timeout are critical because Ansible job logs and large project synchronizations can exceed default Nginx timeout and size limits, leading to 504 Gateway Timeout errors.

Data Protection: Backup and Restore Strategy

Maintaining a backup strategy is paramount since the AWX database contains all the critical logic, credentials, and inventory data. The AWX Operator provides a native AWXBackup custom resource to automate this process.

A backup definition requires a backup PVC and a specific storage class. The following manifest defines a daily backup:

yaml apiVersion: awx.ansible.com/v1beta1 kind: AWXBackup metadata: name: awx-backup-daily namespace: awx spec: deployment_name: awx backup_pvc: awx-backup-claim backup_pvc_namespace: awx backup_storage_class: standard backup_storage_requirements: requests: storage: 10Gi

To initiate the backup process:

bash kubectl apply -f awx-backup.yml

The status of the backup can be monitored using:

bash kubectl -n awx get awxbackup

This backup mechanism ensures that in the event of a cluster failure, the state of the AWX instance can be restored to a known good point, provided the backup PVC is preserved.

Comparison of Deployment Methods

The following table summarizes the differences between the various ways of exposing and deploying AWX on Kubernetes.

Conclusion

The deployment of AWX on Kubernetes via the AWX Operator represents the most maintainable and scalable architecture for Ansible automation. By moving from manual installation to an operator-led model, organizations gain the ability to manage the complex lifecycle of the AWX stack—including the critical PostgreSQL layer—through declarative YAML manifests. The integration of local storage paths for non-cloud environments, the use of Ingress for production-grade traffic management, and the implementation of the AWXBackup resource create a resilient environment capable of supporting enterprise automation.

The success of this deployment relies on strict adherence to resource minimums (4GB RAM/2 CPUs) and a deep understanding of the storage layer. Whether utilizing a simple k3s setup or a massive EKS cluster, the operator pattern ensures that the "desired state" of the automation platform is always maintained, reducing the operational overhead for the DevOps team and providing a stable foundation for the "wonderful world of ansible."

Sources

1. OneUptime Blog: Install AWX on Kubernetes
2. Veeam Community: Install AWX on k3s
3. GitHub: AWX Operator Repository
4. GitHub: AWX Operator Issue 1958