In the architecture of modern CI/CD pipelines, the isolation of execution environments is both a security feature and a logistical challenge. GitHub Actions operates on the principle that each job runs on a separate runner instance, often on entirely different virtual machines. This isolation ensures that a failure in one job does not corrupt the state of another, but it also breaks the conventional Unix assumption that a shell variable persists across commands. When a workflow spans multiple jobs, the default behavior is that no data survives the transition from one runner to the next. To build robust, dynamic pipelines, engineers must implement specific mechanisms to serialize state, transmit it across the network, and deserialize it in downstream contexts. The strategies for achieving this range from native GitHub Artifacts to specialized community actions, each with distinct trade-offs regarding complexity, security, and maintainability.
The Isolation Boundary: Steps vs. Jobs
Understanding the scope of variable visibility is the prerequisite for effective data sharing. In GitHub Actions, the hierarchy of execution is defined as Workflow > Job > Step. A Step is the smallest unit of work, typically a single command or a call to a reusable action. Because all steps within a single job execute on the same runner, they share the same file system and environment context. Writing to the special $GITHUB_ENV file allows a variable to persist from one step to the next within that same job.
However, Jobs are the fundamental units of parallelization and isolation. When a workflow defines multiple jobs, GitHub provisions separate runners for each. Consequently, environment variables set in Job A are completely invisible to Job B. This architectural decision requires explicit data passing mechanisms. The complexity arises because the "next" step might be on the same machine, but the "next" job is almost certainly on a different one. Engineers must choose between the native Step Output model, which is verbose but built-in, or file-based persistence, which is more flexible for complex data structures but requires manual serialization.
Native Mechanisms: GITHUB_OUTPUT and Job Dependencies
The official, native approach to passing data between jobs relies on the $GITHUB_OUTPUT file and job dependencies. This method is structured but can become cumbersome when dealing with multiple variables or dynamic values.
To pass a variable from one job to another, the producer job must write the value to $GITHUB_OUTPUT rather than $GITHUB_ENV. This is critical because $GITHUB_ENV is scoped to the current job's environment, while $GITHUB_OUTPUT is exposed to downstream jobs.
yaml
echo "ENVIRONMENT=dev" >> $GITHUB_OUTPUT
However, a common requirement is to use the variable within the same job that produces it, while also passing it to a subsequent job. Writing to both files manually is redundant. The standard Unix utility tee solves this by duplicating the output stream. Using tee -a (append mode) allows a single command to write to both $GITHUB_ENV and a custom file, or effectively to $GITHUB_ENV and $GITHUB_OUTPUT if structured correctly. In practice, many engineers write to a file and then use tee to append to $GITHUB_ENV and the output file simultaneously.
The consumer job must then declare a dependency on the producer job using needs: and access the output via GitHub's expression syntax.
```yaml
jobs:
producer:
runs-on: ubuntu-latest
outputs:
env-value: ${{ steps.set-var.outputs.ENVIRONMENT }}
steps:
- name: Set variable
id: set-var
run: |
echo "ENVIRONMENT=dev" >> $GITHUB_OUTPUT
consumer:
runs-on: ubuntu-latest
needs: producer
steps:
- name: Read variable
run: |
echo "Received: ${{ needs.producer.outputs.env-value }}"
```
While this method is secure and native, it requires strict adherence to the output schema. Every variable must be explicitly defined in the outputs map of the job. This becomes tedious when managing dozens of dynamic environment variables, leading many engineers to seek more flexible alternatives.
File-Based Persistence with GitHub Artifacts
A more flexible, albeit more complex, approach involves serializing environment variables into a file, uploading that file as a GitHub Artifact, and downloading it in the downstream job. This method treats configuration data as a build artifact, leveraging GitHub's built-in storage infrastructure.
The producer job writes the variables to a file, such as env.vars. The format of this file matters; a simple key-value format (e.g., KEY=VALUE) is standard for shell environments. The tee command is often used to write to both $GITHUB_ENV (for immediate use in the current job) and the file (for downstream transmission).
bash
echo "ENVIRONMENT=$ENVIRONMENT" | tee -a $GITHUB_ENV env.vars
Once the file is created, it is uploaded using the official actions/upload-artifact action. The artifact name should be unique to the workflow run to prevent collisions from concurrent executions. The github.run_id context provides a unique identifier for each workflow run.
yaml
- name: Upload Environment Artifacts
uses: actions/upload-artifact@v3
with:
name: env-cache-${{ github.run_id }}
retention-days: 365
path: env.vars
The downstream job then downloads this artifact and reads its contents into the environment. This is typically done by concatenating the file content into $GITHUB_ENV.
```yaml
- name: Download Environment Artifacts
uses: actions/download-artifact@v3
with:
name: env-cache-${{ github.run_id }}
- name: Set Environment Variables
run: |
cat env.vars >> $GITHUB_ENV
```
This method is powerful because it can handle complex data, binary files, or large sets of variables without cluttering the YAML configuration with explicit output definitions. However, it introduces a critical security constraint: artifacts are readable by anyone with access to the repository. Therefore, sensitive data such as API keys, passwords, or tokens must never be stored in artifacts. This method is best suited for non-sensitive, dynamic configuration data like environment names, build IDs, or deployment targets.
Specialized Actions for Data Persistence
To abstract the complexity of file serialization and artifact management, community-developed actions provide a higher-level interface. One notable example is nick-fields/persist-action-data. This action simplifies the process of persisting data between jobs by handling the artifact upload and download internally.
The action allows users to specify data from step outputs or job outputs and retrieve them in subsequent jobs as environment variables or step outputs. This reduces the boilerplate code required for artifact management.
yaml
- uses: nick-fields/persist-action-data@v1
with:
data: ${{ steps.some-step.output.some-output }}
variable: SOME_STEP_OUTPUT
In a downstream job, the same action can be used to retrieve the variable.
yaml
- uses: nick-fields/persist-action-data@v1
with:
retrieve_variables: SOME_STEP_OUTPUT
This approach encapsulates the logic of uploading to an artifact and downloading it, providing a cleaner API for workflow authors. It is particularly useful when the same data needs to be accessed by multiple jobs or when the data is derived from complex step outputs.
Note that the ownership of this action was transferred from nick-invision to nick-fields in February 2022. Existing workflows referencing the old namespace will continue to function, but it is recommended to update references to the new namespace for clarity and maintenance.
Dynamic Job Context and Environment Configuration
Beyond simple variable passing, data can be used to dynamically configure the execution context of downstream jobs. For instance, a job can determine the deployment environment (e.g., staging vs. production) based on inputs or previous steps and then set the environment field of a subsequent job accordingly.
This is achieved by using job outputs to set the environment property. The context of the second job is not resolved until the first job completes, allowing for runtime decision-making.
```yaml
jobs:
determine-env:
runs-on: ubuntu-latest
outputs:
target-env: ${{ steps.set-env.outputs.ENV }}
steps:
- name: Set Environment
id: set-env
run: echo "ENV=staging" >> $GITHUB_OUTPUT
deploy:
needs: determine-env
environment: ${{ needs.determine-env.outputs.target-env }}
runs-on: ubuntu-latest
steps:
- name: Deploy
run: echo "Deploying to ${{ needs.determine-env.outputs.target-env }}"
```
This pattern is essential for advanced deployment strategies where the target environment is not known at workflow definition time but is determined by the workflow's logic. It allows for a single workflow file to handle multiple environments dynamically, reducing duplication and improving maintainability.
Conclusion
Passing data between GitHub Actions jobs is a nuanced challenge that requires balancing security, simplicity, and flexibility. The native $GITHUB_OUTPUT mechanism is the most secure and integrated approach for simple, well-defined variables, but it lacks the flexibility for complex, dynamic data. File-based persistence with Artifacts offers greater flexibility and is suitable for non-sensitive data, though it requires manual management of serialization and deserialization. Specialized actions like persist-action-data provide a middle ground, abstracting the complexity of artifact management while leveraging the underlying infrastructure. Ultimately, the choice of method depends on the nature of the data, the security requirements, and the complexity of the workflow. Engineers must carefully consider the scope of each variable and the isolation boundaries of their jobs to design robust and efficient CI/CD pipelines.