Architecting Directed Acyclic Graphs with GitLab CI Needs and Modular Include Frameworks

The orchestration of modern software delivery requires more than simple linear execution. In traditional CI/CD pipelines, the stage-based execution model acts as a synchronization barrier; every job in a designated stage must reach a terminal state—typically success—before any job in the subsequent stage can commence. While this provides a predictable flow, it introduces significant latency, especially in complex environments like monorepos or multi-platform build systems where certain tasks are functionally independent of others in the same stage. To overcome these bottlenecks, GitLab CI introduces the needs keyword, transforming a linear sequence into a Directed Acyclic Graph (DAG). This architectural shift allows for the precise definition of job dependencies, enabling jobs to trigger as soon as their specific requirements are met, regardless of the global stage status. When coupled with the include keyword, this capability allows teams to build modular, reusable, and highly scalable pipeline configurations that can be shared across multiple projects and instances.

The Mechanics of the Needs Keyword

The needs keyword is designed to bypass the traditional stage-gate mechanism of GitLab CI. In a standard pipeline, if a project has stages defined as build, test, and deploy, a job in the test stage cannot start until every single job in the build stage has successfully completed. This "all-or-nothing" approach creates a bottleneck where a slow, unrelated build job can delay the start of a fast test job.

By utilizing needs, a developer specifies exactly which jobs must finish before the current job can start. This creates a DAG structure where the flow of execution is determined by dependency rather than by the order of stages.

Core Functional Impacts of Needs

The implementation of needs has several immediate consequences for pipeline performance and execution flow:

• Parallelism Enhancement: In monorepo architectures, independent services can be built and tested in parallel execution paths. A service A test job only needs to wait for service A build job, not for service B or C to finish their respective builds.
• Multi-platform Optimization: When compiling for different operating systems or architectures, a "compile-windows" job and a "test-windows" job can form a direct chain, allowing the Windows test suite to run while the Linux build is still processing.
• Accelerated Feedback Loops: Developers receive error reports and test results significantly faster because the pipeline does not wait for the slowest job in a preceding stage to complete.

Configuration Variations

The needs keyword can be applied in various ways depending on the desired trigger behavior:

• Specific Job Dependencies: Listing one or more jobs that must complete successfully.
• Immediate Execution: Using needs: [] instructs the runner to start the job immediately upon pipeline creation, ignoring all preceding stages and jobs. This is particularly useful for linting or security scans that have no dependencies.

Advanced Dependency Management and the Optional Parameter

A critical challenge in complex pipelines is handling conditional dependencies. In many real-world scenarios, a job may depend on a set of previous jobs, but some of those jobs might be skipped due to rules or only/except clauses. If a job needs a job that was not added to the pipeline, the pipeline will typically fail.

The Optional Attribute

To solve the problem of conditional job execution, GitLab provides the optional: true attribute within the needs array. This attribute allows a job to start even if the specified dependency was not created or was skipped by the pipeline logic.

For example, consider a scenario where a pipeline has two potential dependency jobs: create_dotenv (for main/tag branches) and create_dotenv_test (for feature branches). A subsequent install job needs the output of whichever job actually ran. By configuring the needs section as follows:

yaml install: stage: install script: - ./bootstrap.sh needs: - job: create_dotenv artifacts: true optional: true - job: create_dotenv_test artifacts: true optional: true

This configuration ensures that the install job will proceed as long as at least one of the optional dependencies is met, or even if none are met, preventing the pipeline from crashing due to a missing dependency.

Limitations of Optional Dependencies

While optional: true prevents pipeline failure, it introduces a logic gap. If all dependencies are marked as optional, the dependent job will run even if none of the previous jobs actually executed. This can lead to failures within the script block if the job expects an artifact (like a .env file) that was never created. In such cases, advanced users may need to implement external logic, such as using the GitLab API to cancel jobs or implementing complex shell scripts to validate the existence of required files.

Comparison of Staged Execution vs. DAG Execution

The following table illustrates the fundamental differences between the traditional stage-based approach and the DAG approach enabled by needs.

Modular Pipeline Architecture via Include

To maintain complex DAGs and large sets of jobs, placing everything in a single .gitlab-ci.yml file becomes unmanageable. The include keyword allows for the decomposition of the pipeline into multiple, smaller, and more maintainable YAML files.

Inclusion Methods

GitLab provides several sub-keys under include to source configurations from different locations:

• include:local: Used to reference files within the same project repository. This is the most common method for restructuring a project's CI logic.
• include:template: Used to incorporate official GitLab CI/CD templates. These are sophisticated reference templates provided by GitLab.com that offer "out of the box" functionality for common languages and frameworks.
• include:file: Used to include a specific file from a different project or path.
• include:remote: Used to fetch a YAML file from a remote URL.

Implementation of a Modular Python Pipeline

Consider a scenario where common Python configurations are stored in a shared file located at .gitlab/ci/common.gitlab-ci.yml. This file defines the base environment and shared variables.

```yaml
stages:
- lint
- test
- run
- deploy

default:
interruptible: true

variables:
PYCOLORS: "1"
CACHEPATH: "$CIPROJECTDIR/.cache"
UVCACHEDIR: "$CACHEPATH/uv"
UVPROJECTENVIRONMENT: "$CACHEPATH/venv"
PIPCACHEDIR: "$CACHE_PATH/pip"

.base_image:
image: python:3.13

.baserules:
rules:
- if: $CICOMMITBRANCH == $CIDEFAULT_BRANCH

.dependencies:
beforescript:
- pip install --upgrade pip
- pip install uv
- uv sync --frozen
cache:
key:
files:
- uv.lock
prefix: $CIJOBIMAGE
paths:
- "$CACHEPATH"
```

This shared configuration can then be brought into the primary .gitlab-ci.yml using the include keyword, allowing the main file to remain clean and focused only on the high-level orchestration.

Handling Complex Conditional Dependencies in Practice

In professional DevOps environments, developers often encounter "hybrid" needs—situations where they want the benefits of a DAG but still rely on stage-based ordering for specific subsets of the pipeline.

The Dotenv Artifact Problem

A common requirement is the use of dotenv artifacts to pass variables between jobs. When using needs, the artifacts: true property must be explicitly set to ensure the dependent job downloads the required environment files.

If a user has a setup where a job must run after either job_a OR job_b, and both these jobs are conditional (one for main branch, one for test branch), the following logic is applied:

1. Define the jobs using extends to share common logic.
2. Apply rules or only/except to ensure only one of the two "provider" jobs runs.
3. Use needs with optional: true in the "consumer" job.

This prevents the pipeline from failing when one of the provider jobs is absent, while still allowing the consumer job to pick up the dotenv artifact from whichever provider actually executed.

Dealing with Resource Contention

When using needs to trigger jobs early, there is a risk of creating resource contention, especially when interacting with external infrastructure like AWS EKS. To prevent multiple jobs from attempting to modify the same cluster simultaneously, the resource_group keyword should be used. This ensures that jobs are executed non-concurrently, even if the DAG structure would otherwise allow them to start at the same time.

Example of a resource-protected job:

yaml make-eks: stage: create-management-cluster tags: - shell resource_group: pipeline_mutex script: - make eks

Analysis of Pipeline Restructuring and Maintenance

The transition from a monolithic .gitlab-ci.yml to a distributed structure using include and a dependency-driven flow using needs represents a maturity shift in CI/CD design.

The primary advantage of this approach is the reduction of cognitive load. By splitting the pipeline into functional files (e.g., lint.yml, test.yml, deploy.yml), developers can isolate changes to specific parts of the pipeline without risking the integrity of the entire configuration. Furthermore, the use of official GitLab templates allows organizations to adopt industry-standard practices without reinventing the wheel, though it is noted that these templates are subject to change over time, requiring periodic audits.

The move toward DAGs via needs effectively eliminates "dead time" in the pipeline. In a traditional stage-based pipeline, the total time to completion is the sum of the slowest job in each stage. In a DAG-based pipeline, the total time is the sum of the slowest path (the critical path) from the start job to the end job. This optimization is critical for maintaining high developer velocity in large-scale projects.

Sources

1. hifis.net - Using Includes
2. GitLab Forum - OR condition with needs
3. GitLab Documentation - needs