The ability to execute tasks in parallel is one of the most critical performance levers in Ansible. Parallelism in the context of software refers to the capacity of a system to spawn multiple processes to execute tasks in tandem. In the Ansible ecosystem, this capability manifests as the ability to interact with numerous managed hosts simultaneously rather than sequentially. As an organization's infrastructure scales from a handful of servers to thousands of nodes, the efficiency of this parallel execution becomes the primary determinant of deployment speed and operational agility. Rather than processing a single host at a time, Ansible spawns individual processes for each host it manages, based on the specific configuration defined by the administrator. This architecture ensures that the control node can push configurations or execute commands across a vast fleet of servers with minimal linear time increase.
The Mechanics of Ansible Forks
In Ansible, the individual parallel processes used to manage hosts are known as forks. A fork represents a single worker process on the control node that handles the connection and execution of tasks for one managed node. By default, Ansible is configured to use five forks. This means that even if a playbook targets one hundred servers, Ansible will only process them in batches of five at a time.
The relationship between forks and system resources is direct and proportional. Increasing the number of forks allows for greater parallelism, but it simultaneously increases the consumption of resources on the Ansible control node. The primary resource impacted is memory, as each fork requires its own overhead to maintain the connection and track the state of the task.
For administrators operating in environments with a large number of managed nodes and a control node equipped with substantial memory and CPU resources, it is common to increase the fork count to 50 or 100. This ensures that the "bottleneck" is shifted from the control node's process limit to the network capacity or the remote hosts' responsiveness.
It is important to note that Ansible is designed to be efficient with process spawning. If the configuration is set to 50 forks, but the current playbook execution only targets 12 hosts, Ansible will only spawn 12 processes. It does not waste resources by creating idle forks that have no target hosts to manage.
The configuration of forks can be managed through two primary methods:
- Persistent Configuration: Editing the
ansible.cfgfile allows an administrator to set a global default for all executions. This is ideal for environments where the control node has consistent, high-spec hardware. - Dynamic Command-Line Override: The
-fflag can be used during the execution of anansiblead-hoc command or anansible-playbookcommand. This provides the flexibility to adjust parallelism on a per-command basis, allowing the user to set a higher value for large groups or a lower value for sensitive production clusters.
Execution Strategies and Host Parallelism
By default, Ansible employs a strategy where each playbook task is run on each host at the same time, subject to the fork limit. The playbook does not proceed to the next task until every host in the current batch has completed the current task. This means that while tasks are running in parallel across different hosts, the sequence of tasks within the playbook remains synchronized.
To refine this behavior, Ansible provides several strategies:
The Free Strategy
The "free" strategy allows Ansible to execute tasks on each host as fast as possible without waiting for other hosts. In the default linear strategy, if one host is slow to respond to a task, all other hosts must wait for that slow host to finish before they can move to the next task in the playbook. The free strategy eliminates this synchronization point, allowing faster hosts to race ahead to subsequent tasks.
The Serial Keyword and Batching
The serial keyword is used to limit playbook execution to a specific batch size. This is particularly useful for rolling updates where you cannot afford to take down all servers at once. By defining a batch size, you ensure that only a subset of hosts is updated at any given time, maintaining service availability.
Refined Execution Controls
Beyond basic strategies, Ansible offers more granular controls to manage how tasks are dispatched:
- Throttle: This allows the user to limit the number of parallel executions for a specific task, regardless of the global fork setting.
- Run_once: This ensures a task is executed only once in a play, even if the play targets multiple hosts. This is critical for tasks like database migrations or creating a shared resource where repeating the action across all hosts would cause errors or duplication.
Parallelism via Asynchronous Tasks
While forks handle parallelism across different hosts, there is a different requirement when one needs to run a task several times in parallel on a single host or across a set of hosts using a loop. By default, a task utilizing a loop (such as with_items) is executed serially. This means the loop completes for the first item before starting the second.
This becomes a significant performance bottleneck when dealing with cloud provider APIs, such as Amazon Web Services (AWS). For example, creating EC2 instances or RDS databases are "long-running" operations. If a loop is used to create ten instances, the total time taken is the sum of the creation time for each individual instance.
To solve this, Ansible provides the async keyword. When combined with poll: 0, Ansible can "fire and forget" the tasks.
The following example demonstrates how to run tasks in parallel and wait for them to finish:
yaml
- name: Run tasks in parallel
hosts: localhost
connection: local
gather_facts: no
tasks:
- name: Pretend to create instances
command: "sleep {{ item }}"
with_items:
- 6
- 8
- 7
register: _create_instances
async: 600
In this configuration, async: 600 defines the maximum runtime in seconds. Because these tasks are launched asynchronously, they run in parallel rather than waiting for the sleep command of the first item to finish before starting the second.
Overcoming the Limitations of Roles and Tasks in Parallel
A known limitation in Ansible is that the async mode does not function when using include_role or include_tasks within a loop. Attempting to use poll with include_role results in a fatal error: ERROR! 'poll' is not a valid attribute for a IncludeRole.
To execute roles in parallel on the same host, a technical workaround involving "pseudo-hosts" is required. Since the localhost IP range (127.0.0.1 through 127.255.255.254) all refer to the local machine, an administrator can create a dynamic host-group at runtime using these different loopback addresses.
By assigning each item in a loop to a different localhost IP, Ansible perceives them as different hosts and applies its native host-level parallelism (forks) to the roles.
The logic follows this pattern:
- User 1 (Dave) is assigned to 127.0.0.1
- User 2 (Eva) is assigned to 127.0.0.2
- User 3 (Hans) is assigned to 127.0.0.3
If these roles must interact with a remote server, the delegate_to keyword can be used to redirect the execution to the actual remote host while still benefiting from the parallel dispatch initiated by the pseudo-localhosts.
Comparing Execution Methods
The following table summarizes the different methods of achieving parallelism within Ansible.
| Method | Level of Parallelism | Primary Control | Use Case |
|---|---|---|---|
| Forks | Across Hosts | -f or ansible.cfg |
Scaling execution across many servers |
| Async | Within Loops | async / poll |
Long-running API calls (AWS/GCP) |
| Serial | Batch-based | serial: keyword |
Rolling updates, zero-downtime deploys |
| Pseudo-Hosts | Role-level | Localhost IP range | Running multiple roles in parallel on one host |
| Free Strategy | Task-level | strategy: free |
Maximum speed, ignoring task synchronization |
Executing Multiple Playbooks in Parallel
While standard Ansible execution focuses on parallelizing tasks across hosts within a single playbook, there are scenarios where multiple, entirely different playbooks need to run simultaneously. This cannot be achieved using a single ansible-playbook command, as that process is sequential by nature.
To achieve this, external tools or custom wrappers like ansible-parallel can be used. This allows the execution of different YAML files at the same time.
Example command for running two playbooks in parallel:
bash
ansible-parallel tasks-normal-fixed.yml tasks-normal-random.yml
In this scenario, the output will show the completion of both playbooks independently. This is the most efficient way to handle disjointed automation workflows that do not depend on each other's output.
Resource Considerations and Performance Tuning
The degree of parallelism should never be set to a maximum value without considering the underlying infrastructure. The "optimal" fork count is a balance between the control node's capacity and the target environment's limits.
- Network Bandwidth: High parallelism can saturate the network interface of the control node, leading to packet loss or timeout errors.
- Disk I/O: If tasks involve heavy file transfers or backups, too many parallel processes can cause disk contention.
- Target Load: Spawning 100 parallel connections to a single target (in the case of pseudo-hosts) can overload the SSH daemon or the application being configured.
If a process that normally takes 32 seconds with forks=1 is shifted to a higher fork count, the time should drop significantly, provided the hardware can support the concurrency.
Conclusion
The mastery of parallelism in Ansible requires a tiered approach, moving from global fork management to specific task-level asynchronous strategies. The default behavior of Ansible provides a strong foundation for host-level parallelism, but the introduction of the serial keyword and the free strategy allows for the precision required in enterprise-grade rolling updates. For advanced users, the use of async and poll solves the problem of long-running external API calls, while the pseudo-localhost technique bypasses the inherent limitations of include_role. Ultimately, the goal of increasing parallelism is to reduce the "time-to-completion" for infrastructure changes, but this must always be balanced against the resource constraints of the control node to avoid systemic failure.