GitLab Detached Pipeline Anomalies and Merge Request Orchestration

The architectural complexity of GitLab CI/CD pipelines often introduces subtle but critical behaviors regarding how pipelines are instantiated during the Merge Request (MR) lifecycle. One of the most perplexing phenomena encountered by DevOps engineers is the emergence of detached pipelines—specifically those that appear as "mutilated" or redundant executions alongside standard branch pipelines. This phenomenon typically arises from the intersection of rules:if logic and the specific triggers associated with merge_request_event. When a pipeline is categorized as a merge request pipeline, it operates on the source branch but is logically detached from the standard branch push flow, creating a distinct execution context. However, bugs in the orchestration layer can lead to a state where both a regular branch pipeline and a detached MR pipeline are triggered simultaneously, often resulting in the omission of critical jobs that lack explicit rule definitions.

The Mechanics of Merge Request Pipelines

A merge request pipeline is a specialized execution flow designed to run every time changes are committed to a source branch within an active merge request. Unlike standard pipelines that trigger on every push to any branch, these pipelines are explicitly tied to the MR entity.

The conditions that trigger these pipelines include:

The creation of a new merge request from a source branch that contains one or more commits.
The act of pushing a new commit to the source branch after an MR has already been opened.
The manual invocation of a pipeline via the Pipelines tab within the merge request interface.

These pipelines possess a unique operational characteristic: they run exclusively on the contents of the source branch, purposefully ignoring the content of the target branch. This ensures that the tests and validations are performed on the proposed changes before the merge occurs. In the GitLab interface, these are easily identifiable by the presence of a merge request label.

To successfully implement this behavior, the .gitlab-ci.yml configuration must meet specific prerequisites:

The configuration must utilize job rules or workflow rules that explicitly match the condition CI_PIPELINE_SOURCE == "merge_request_event".
The user initiating the pipeline must possess a role of Developer, Maintainer, or Owner for the source project.
The repository must be hosted natively on GitLab; external mirrors or external repositories are not supported for this specific pipeline type.

It is critical to note that rules defined within include: blocks, such as those using include:component, do not satisfy the requirement for triggering merge request pipelines. This means that the logic governing the merge_request_event must be present in the main configuration or a compatible included file to be recognized by the GitLab orchestrator.

The Detached Pipeline Bug: Rule-Based Mutilation

A significant issue has been identified where the use of rules:if causes the simultaneous execution of two pipelines upon a force push to an MR: one regular branch pipeline and one detached MR pipeline. This redundancy is not merely an issue of wasted compute resources but leads to a catastrophic failure of the pipeline structure known as "job mutilation."

In these scenarios, the detached pipeline often only includes jobs that have explicit rules: defined. Any job that does not have a rules: block—even if it is a critical dependency for a later stage—is completely omitted from the detached pipeline.

This creates a severe impact on pipeline stability. If a developer has a complex pipeline where a deploy job depends on a build job, and only the deploy job has rules: (e.g., to allow manual triggering in a demo lab), the build job (which lacks rules) will be missing from the detached pipeline. Consequently, the pipeline fails because the unmet dependency cannot be satisfied.

The logic failure manifests as follows:

The system recognizes the rules:if condition and triggers the MR pipeline.
The system simultaneously triggers a branch pipeline due to the push event.
The detached MR pipeline fails to inherit jobs that are "global" (those without specific rules), stripping the pipeline of its foundational tasks.

This bug effectively breaks the utility of rules: in complex pipelines with inter-job dependencies, forcing engineers to either duplicate jobs or revert to the older only/except syntax to maintain stability.

Runner Behavior and Detached HEAD States

The interaction between the GitLab Runner and the repository state during these pipelines can lead to further complications, particularly regarding "detached HEAD" states and git merge failures.

In certain runner versions, such as Runner 13.1.1, the checkout process results in a detached HEAD. This means the runner is not on a local branch but is pointing directly to a specific commit hash. When a runner attempts to perform a coherence check or merge the current branch with a master branch, it may encounter a fatal error: fatal: refusing to merge unrelated histories.

The difference in behavior between runner versions is evident in the logs:

Version 12.9.0: The logs show the runner explicitly checking out the commit as a specific branch (e.g., Checking out 16b0a39c as 20-branchname), and it performs a fetch of the remote refs.
Version 13.1.1: The logs show a direct checkout of the commit hash (e.g., Checking out 7410f5aa as 20-branchname) without the same level of branch-awareness, leading to the detached HEAD state.

This is particularly problematic for jobs that require git operations, such as the "Coherence Check" jobs, which may use templates to determine manual triggers based on whether a merge request is marked as "WIP" (Work In Progress) via the title.

Pipeline Type Classification in Omnibus GitLab

To avoid the chaos of overlapping pipeline triggers and detached states, the omnibus-gitlab project implements a more rigid architectural approach. Instead of relying solely on generic rules, it utilizes a specific variable named PIPELINE_TYPE to categorize and attach jobs to specific pipeline contexts.

The following table outlines the specialized pipeline types used in the Omnibus ecosystem:

Pipeline Type	Mirror(s)	Purpose and Requirements
`DEPENDENCY_SCANNING_PIPELINE`	Canonical	Security vulnerability scans; requires `DEPENDENCY_SCANNING` variable.
`LICENSE_PAGE_UPDATE_PIPELINE`	Canonical	License page updates; requires `PAGES_UPDATE` variable.
`CACHE_UPDATE_PIPELINE`	Canonical, QA	Gem and package build cache updates; requires `CACHE_UPDATE` variable.
`DURATION_PLOTTER_PIPELINE`	QA	Package builds for plotting duration; requires `DURATION_PLOTTER` variable.
`DOCS_PIPELINE`	Canonical, Security	Documentation pipelines; requires `docs-` prefix or `-docs` suffix in branch name.
`PROTECTED_TEST_PIPELINE`	Canonical, Security	Runs on protected branches/tags; excludes trigger jobs and `danger-review`.
`GITLAB_BRANCH_TEST_PIPELINE`	Canonical, Security	Runs on non-protected branch pushes; includes trigger jobs, excludes `danger-review`.
`GITLAB_MR_PIPELINE`	Canonical, Security	Specifically for MRs within Canonical/Security; includes trigger jobs.

This structured approach ensures that jobs are not "mutilated" by the orchestrator, as each job is explicitly mapped to a PIPELINE_TYPE rather than relying on the volatile interaction between rules:if and the merge_request_event.

Specialized Execution Contexts and Mirroring

Beyond the standard detached pipelines, GitLab's infrastructure supports complex mirroring and triggering mechanisms that move beyond simple commit-based triggers.

The GitLab QA Mirror serves as a critical environment where pipelines are passed Docker images created by specific jobs (such as Trigger:gitlab-docker) and the GitLab Rails pipeline. This ensures that the entire test suite is executed using a consistent set of images across different environments.

Further specialization is seen in the following contexts:

RAT (Regression Analysis Tool): A manual job that triggers a pipeline in the RAT project. This process involves passing the URL of a package built by the Trigger:package job, which then spins up a PostgreSQL HA instance via GET to run QA tests. This is restricted to the QA mirror on triggered EE (Enterprise Edition) pipelines.
OS-Specific Branching: Jobs like <OS_NAME>-branch are used to build packages for specific operating systems, which are then pushed to S3 buckets. These are restricted to Release mirrors on branch and nightly pipelines.
Hardware Restrictions: Certain jobs are restricted by the edition of GitLab; for example, Raspberry Pi jobs run exclusively on CE (Community Edition) branches, while SLES (SUSE Linux Enterprise Server) jobs run only on EE branches.
Containerization: The Docker-branch job specifically builds a GitLab Docker image using the package generated by the Ubuntu 24.04-branch job.

Analysis of Pipeline Blocking and Rule Conflicts

A recurring failure mode in the use of environments and review apps is the "blocked pipeline" scenario. This occurs when users transition from the when keyword to the rules keyword in their .gitlab-ci.yml configuration.

The expectation is that the pipeline should automatically stop or transition based on the rules provided. However, in practice, the pipeline can become blocked. This typically happens because the rules logic creates a state where no jobs are eligible to run, or the conditions for the environment's "stop" action are not met due to a mismatch in the rules:if logic.

When using rules, the pipeline's behavior changes fundamentally:

when is a simple attribute of a job.
rules is a complex logic engine that can change the when status of a job from manual to on_success based on environment variables.

If a user attempts to use rules to manage the lifecycle of a review app, the detached nature of the MR pipeline can interfere with the ability of GitLab to track the "stop" action, effectively leaving orphaned environments or blocked pipelines that prevent further merges.

Conclusion: The Interplay of State and Orchestration

The phenomenon of the GitLab detached pipeline is a symptom of the tension between branch-based execution and merge-request-based orchestration. When a pipeline is "detached," it is essentially operating in a virtualized state that represents the intent of the merge rather than the raw state of the branch.

The "mutilation" of pipelines during these events highlights a critical flaw in how GitLab handles jobs without explicit rules during an MR event. By ignoring jobs that lack rules: blocks, the orchestrator breaks the dependency chain (needs/dependencies), rendering the pipeline unusable for complex software delivery.

To mitigate these issues, organizations must move away from implicit rule assumptions. The omnibus-gitlab model of using a PIPELINE_TYPE variable is the most robust solution, as it replaces the ambiguous merge_request_event trigger with an explicit identity for every pipeline. Without this explicit mapping, developers are left fighting a battle against detached HEADs, missing jobs, and redundant pipeline executions.

The evolution of the GitLab Runner from version 12.9.0 to 13.1.1 further illustrates this instability, where the shift toward a more "pure" commit checkout (detached HEAD) broke the ability of many scripts to perform coherence checks against the master branch. This necessitates a deeper understanding of git internals when writing before_script sections for complex CI/CD workflows.