In continuous integration and deployment pipelines, the exit code of a process is the primary signal for success or failure. By convention, an exit code of zero indicates success, while any non-zero value signals an error. This binary logic is foundational to shell scripting and process management, but it often proves too rigid for complex automation workflows. Engineers frequently encounter scenarios where a non-zero exit code does not represent a catastrophic failure but rather a specific state, a warning, or an expected outcome that requires a different handling strategy. Whether managing GitHub Actions runners, Concourse CI pipelines, or systemd services, the inability to distinguish between "clean shutdown with status 1" and "unrecoverable crash" leads to broken builds and false negatives. This article explores the mechanisms available to suppress, reinterpret, or manage these exit codes, focusing on GitHub Actions as the primary context, while drawing parallels from other systems like systemd and Concourse.
The Mechanics of Exit Code Interpretation in GitHub Actions
When a GitHub Actions runner executes a shell script, it relies on the exit code returned by the process to determine the step's outcome. If a script exits with a non-zero code, the runner automatically annotates the log with a message such as "Process completed with exit code N." This annotation, while informative, can be redundant if the script itself has already provided detailed error output or if the exit code was intentional. For instance, a user might want to use actions/github-script with core.setFailed to handle failures programmatically, avoiding the default ScriptHandler behavior that generates these redundant annotations. Ideally, engineers prefer to keep their logic in standard bash scripts rather than migrating to JavaScript-based actions, but the runner's default behavior often forces a choice between verbose logging and precise control.
The core issue is that GitHub Actions treats every non-zero exit code as a failure unless explicitly told otherwise. This behavior is not unique to GitHub; it is a standard convention in Unix-like systems. However, in the context of automated workflows, this rigidity can be problematic. For example, git diff can emit different exit codes to signify different states, such as "no changes" versus "changes detected." In a build workflow, a high return code might indicate that no changes were found, which is a valid and expected outcome, not a failure. If the workflow is not configured to interpret this correctly, the entire run is marked as failed, leading to unnecessary alerts and wasted developer time.
Suppressing Failures with continue-on-error
The most direct way to prevent a non-zero exit code from failing a job step in GitHub Actions is to use the continue-on-error directive. This setting can be applied at the job or step level. When continue-on-error: true is set, the runner will ignore the exit code of the step and proceed to the next step or job, regardless of whether the previous command succeeded or failed. This is particularly useful for commands that are best-effort or where failure is acceptable.
For example, in a build workflow, a developer might intentionally trigger a step that returns a high exit code to signify that no changes were detected. Without continue-on-error, this would cause the entire workflow run to be marked as failed. By adding continue-on-error: true, the workflow continues successfully, and the exit code is treated as a warning rather than an error. This approach is simple and effective for steps where the outcome is not critical to the overall success of the pipeline.
However, continue-on-error has limitations. It does not allow for conditional logic based on the specific exit code. It simply prevents the failure from propagating. If the exit code is used to control subsequent steps, this blanket suppression might lead to incorrect behavior. Additionally, if a step fails for an unexpected reason, continue-on-error might mask the issue, leading to silent failures that are difficult to debug. Therefore, it should be used judiciously and only for steps where failure is explicitly acceptable.
Handling Complex Logic: Conditional Jobs and Fixing Failures
In more complex scenarios, a non-zero exit code might trigger a remediation process. For instance, a workflow might run a linter like black --check . to ensure Python code is formatted correctly. If the check fails (exit code 1), the developer might want to automatically reformat the code and commit the changes, rather than failing the build. This requires a more sophisticated approach than continue-on-error.
One common pattern is to use multiple jobs with conditional execution. The first job, build, runs the check. If it fails, a second job, reformat, is triggered to fix the issue. The if condition for the reformat job can be set to always() && (needs.build.result == 'failure'). This ensures that the reformat job only runs if the build job failed. Inside the reformat job, the code is reformatted, committed, and pushed back to the repository.
However, this approach has a significant drawback: the final result of the workflow is still marked as a failure, even if the reformat job succeeds. This is because the initial failure in the build job propagates up to the workflow level. The workflow status is determined by the status of the first failing job, regardless of subsequent fixes. This can be misleading, as the pipeline ultimately succeeded in fixing the issue, but the dashboard shows a failure. To address this, some developers attempt to unset the failure code in composite actions, but GitHub has closed issues requesting this feature as "not planned," indicating that there is no built-in way to reset the failure status of a previous job within the same workflow.
yaml
name: autoblack
on: [pull_request, push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Set up Python 3.8
uses: actions/[email protected]
with:
python-version: '3.8'
- name: Install Black
run: pip3 install git+git://github.com/psf/black
- name: Run black --check .
run: black --check .
reformat:
runs-on: ubuntu-latest
needs: [build]
if: always() && (needs.build.result == 'failure')
steps:
- uses: actions/[email protected]
- name: Set up Python 3.8
uses: actions/[email protected]
with:
python-version: '3.8'
- name: Install Black
run: pip3 install git+git://github.com/psf/black
- name: If needed, commit black changes to the pull request
env:
NEEDS_CONTEXT: ${{ toJSON(needs) }}
run: |
black --fast .
git config --global user.name 'autoblack'
git config --global user.email '[email protected]'
git remote set-url origin https://x-access-token:${{ secrets.GITHUB_TOKEN }}@github.com/$GITHUB_REPOSITORY
git checkout $GITHUB_HEAD_REF
echo "$NEEDS_CONTEXT"
git commit -am "fixup: Format Python code with Black"
git push
echo "$NEEDS_CONTEXT"
In this example, even if the reformat job successfully pushes the changes, the workflow result remains failure because the build job failed. This highlights a fundamental limitation in GitHub Actions: the inability to retroactively change the status of a failed job. Workarounds often involve running the check and fix in the same job, using continue-on-error for the check, and then conditionally running the fix if the check failed. This keeps the logic within a single job, allowing the final exit code to be controlled by the fix step.
Parallels in Other Systems: Concourse and Systemd
The challenge of managing non-zero exit codes is not unique to GitHub Actions. Other CI/CD systems and service managers face similar issues. In Concourse CI, for example, there is no built-in way to store state between jobs or pipelines. To track deployment versions, teams often use a dedicated git repository. However, if a deployment job runs twice in a row with no changes, the git push operation will fail because there is nothing to push. This causes the pipeline to fail, even though no actual error occurred.
To address this, Concourse users have requested a parameter similar to git's quiet flag, which suppresses the non-zero exit code on empty pushes. Without this, developers must script around the issue, adding complexity to their pipelines. This is analogous to the GitHub Actions scenario where git diff returns a non-zero code for "no changes," requiring continue-on-error or custom logic to handle.
Similarly, systemd, the system and service manager for Linux, has faced criticism for its handling of exit codes during service stops. Some services, such as Hashicorp Consul, exit with status code 1 even on a clean requested shutdown. This causes systemctl stop to return a non-zero exit code, which can break scripts that expect a zero exit code on success. Systemd users have requested a StopSuccessExitStatus directive, similar to the existing SuccessExitStatus, to allow specific exit codes to be treated as success during the stop phase. This would enable scripts to correctly identify when a service has stopped cleanly, regardless of the exit code it returns.
These examples from Concourse and systemd illustrate a broader trend: the need for more granular control over exit code interpretation. Default behavior often assumes that non-zero equals failure, but real-world scenarios are more nuanced. Whether it's a git push with no changes, a service stopping cleanly with a non-zero code, or a linter finding formatting issues, the ability to customize exit code handling is essential for robust automation.
Composite Actions and the Limits of Control
A specific area of interest in GitHub Actions is the behavior of composite actions. Composite actions allow developers to bundle multiple steps into a single reusable action. However, when a step within a composite action fails, the failure propagates to the composite action, and there is no built-in way to unset or override this failure. Issues on the GitHub Docs repository have requested a section explaining how to set a zero exit code in composite actions when an intermediate step fails, but these requests have been closed as "not planned."
This limitation means that if a composite action contains a step that fails (e.g., actions/download-artifact fails because an artifact is missing), the entire composite action will fail. Even if the failure is expected and handled within the action, the overall status remains failed. This can be problematic for workflows that rely on composite actions for modular logic. Developers must work around this by ensuring that all steps within the composite action either succeed or use continue-on-error to prevent failures from propagating.
The lack of a way to reset the failure status in composite actions mirrors the limitation in multi-job workflows. It suggests that GitHub's design philosophy prioritizes explicit failure signaling over flexible error recovery. While this ensures that errors are visible, it can make it difficult to build resilient workflows that handle expected failures gracefully.
Practical Strategies for Exit Code Management
Given these limitations, developers must adopt practical strategies to manage exit codes effectively. The following approaches can help mitigate the issues discussed:
- Use
continue-on-errorstrategically: Apply this to steps where failure is acceptable or expected, such as best-effort checks or non-critical operations. - Combine check and fix in a single job: Instead of using multiple jobs, run the check and fix in the same job. Use
continue-on-errorfor the check, and then conditionally run the fix if the check failed. This ensures that the final exit code of the job can be controlled. - Script around git issues: For systems like Concourse or GitHub Actions where git operations might fail due to no changes, use the
git push --quietflag or check for changes before pushing. - Handle systemd service stops carefully: If using systemd to manage services that exit with non-zero codes on stop, use wrapper scripts that interpret the exit code correctly or request features like
StopSuccessExitStatusfrom upstream maintainers. - Avoid relying on composite actions for critical logic: If a composite action contains steps that might fail, ensure that the failure is handled within the action itself, using
continue-on-erroror conditional logic.
By adopting these strategies, developers can build more resilient and predictable workflows, even in the face of rigid exit code conventions.
Conclusion
The management of non-zero exit codes in CI/CD pipelines is a nuanced challenge that requires a deep understanding of both the tools and the underlying Unix conventions. GitHub Actions, like other systems, defaults to treating any non-zero exit code as a failure, which can lead to false negatives and broken workflows in complex scenarios. While continue-on-error provides a simple way to suppress failures, it lacks the granularity needed for more sophisticated error handling. The inability to reset failure statuses in composite actions or multi-job workflows further complicates matters, forcing developers to adopt workarounds like combining check and fix steps in a single job.
Looking forward, the industry is slowly recognizing the need for more flexible exit code handling. Requests for features like StopSuccessExitStatus in systemd and quiet push options in Concourse reflect a growing desire for systems that can distinguish between "expected non-zero exit codes" and "true failures." Until these features are widely adopted, developers must rely on careful scripting and strategic use of existing directives to build resilient automation. The key is to anticipate where exit codes might misrepresent the actual state of the system and to design workflows that can interpret them correctly.