Orchestrating GitLab Infrastructure and CI/CD Runners on Amazon EC2

The deployment of GitLab and its associated CI/CD runners on Amazon Elastic Compute Cloud (EC2) represents a sophisticated intersection of version control, automation, and cloud infrastructure. At its core, GitLab CI serves as a critical engine for enterprises to automate their Continuous Integration, Continuous Delivery, and Deployment (CI/CD) processes. This automation is fundamentally driven by two primary components: the .gitlab-ci.yml file, which acts as the blueprint describing the pipeline's jobs, and the GitLab Runner, the actual application responsible for executing those jobs. For organizations managing hundreds of pipelines across diverse environments, manual setup is an untenable strategy. The complexity of provisioning infrastructure, installing the requisite software for various workloads, and configuring runners necessitates a transition toward Infrastructure-as-Code (IaC). By leveraging IaC, engineers can deploy the entire GitLab Runner architecture through scripts, ensuring that every deployment is repeatable, consistent, and easily trackable. This approach not only enforces architectural guardrails and best practices via code but also enables the implementation of autoscaling, which allows enterprises to optimize costs by terminating resources when they are not actively processing jobs.

Architectural Components and AWS Service Integration

Deploying GitLab on AWS requires a coordinated effort across multiple services to ensure high availability, scalability, and security. The architectural footprint extends beyond simple compute instances, integrating storage, database, and networking layers.

The following table details the AWS services utilized in a standard GitLab deployment and their specific roles:

The integration of these services ensures that the application layer remains decoupled from the data layer. For instance, utilizing Amazon RDS for PostgreSQL ensures that database management, patching, and backups are handled by AWS, reducing the operational burden on the DevOps team. Similarly, the use of Amazon S3 for artifacts and LFS objects ensures that the EC2 root volume remains lean and focused on application performance rather than bulk data storage.

Network Foundation and Security Configuration

A robust deployment begins with a carefully planned network topology to isolate traffic and protect sensitive data.

The creation of a Virtual Private Cloud (VPC) is the first step in establishing this environment. A recommended configuration involves creating a VPC named gitlab-vpc with an IPv4 CIDR block of 10.0.0.0/16. This provides a private networking space that is isolated from other AWS accounts. It is critical to navigate to the VPC settings and enable DNS resolution to ensure that internal service discovery functions correctly across the infrastructure.

Following the VPC creation, subnets must be established across at least two Availability Zones (AZs) to ensure high availability. This multi-AZ strategy prevents a single point of failure; if one data center experiences an outage, the traffic can be routed to the other AZ. Public subnets within this architecture require a Route Table and an associated Internet Gateway to allow outbound and inbound traffic from the public internet.

Security is further hardened by the implementation of the AWS Instance Metadata Service Version 2 (IMDSv2). GitLab is designed to support IMDSv2 automatically, falling back to IMDSv1 only if necessary. Requiring IMDSv2 on all EC2 instances is a recommended security best practice to mitigate risks associated with SSRF (Server-Side Request Forgery) attacks.

IAM Roles and S3 Access Control

To avoid the security risk of embedding hard-coded AWS access keys within the GitLab configuration files, the deployment utilizes IAM Roles. This ensures that the EC2 instances possess the necessary permissions to interact with S3 buckets without exposing credentials.

The process for establishing this secure access involves the following steps:

• Create a policy named gl-s3-policy that defines the specific permissions required for S3 interaction.
• Create an IAM role named GitLabS3Access using the EC2 use case.
• Attach the gl-s3-policy to this role.
• Assign this role to the EC2 instances during the launch process.

This configuration allows the GitLab instance to perform read, write, and list operations on S3 buckets, which is essential for managing backups and pipeline artifacts.

GitLab Runner Deployment via Infrastructure-as-Code

Deploying GitLab Runners is often a time-consuming process if done manually. To solve this, the use of IaC allows for the rapid and consistent deployment of runners.

The deployment process utilizes a custom Amazon Machine Image (AMI). To create this, an instance is configured, and then the "Create image" action is selected from the EC2 dashboard. This image, typically named GitLab-Source, serves as the gold image for all future runners.

Once the AMI is ready, a launch template is created. The configuration for the gitlab-launch-template includes:

• AMI Selection: The custom GitLab-Source AMI.
• Instance Type: A minimum of c5.2xlarge is recommended to handle the computational demands of CI/CD workloads.
• Key Pair: A new key pair named gitlab-launch-template with the associated .pem file saved for secure SSH access.
• Storage: The default root volume is 8 GiB, which is sufficient since application data is stored externally in S3 and RDS.

By placing these runners inside an Auto Scaling Group (ASG), the system can automatically scale the number of runners based on the current workload. This prevents pipeline bottlenecks during peak hours and eliminates unnecessary costs during idle periods by terminating unused instances.

Advanced Pipeline Deployment to EC2

GitLab provides specialized templates to facilitate the deployment of applications directly to EC2 instances. The template AWS/CF-Provision-and-Deploy-EC2 automates the bridge between the CI pipeline and the AWS cloud.

The deployment workflow follows a specific sequence of operations:

1. Infrastructure Provisioning: The pipeline uses the AWS CloudFormation API to create a stack based on a JSON object.
2. Artifact Handling: Once the build job completes, the resulting artifact is pushed to an AWS S3 bucket.
3. Application Deployment: The content is then deployed from S3 onto the EC2 instance.

To implement this, the user must provide a JSON configuration for the S3 push, which includes the applicationName, the source (the location where the build job created the artifact), and the s3Location (the destination bucket).

The underlying mechanism for these actions is supported by specific scripts:

• gl-cloudprovision create-stack: This script invokes aws cloudformation create-stack and polls the AWS API until the stack setup is either complete or has failed.
• gl-ec2 push-to-s3: This script handles the transfer of the build artifact to the S3 bucket.
• gl-ec2 deploy-to-ec2: This script utilizes aws deploy create-deployment to push the application to the EC2 instance and polls for success or failure.

ECS Integration and Configuration Constraints

For those utilizing Amazon ECS instead of raw EC2, GitLab provides the AWS/Deploy-ECS.gitlab-ci.yml template. This template is a composite that includes Jobs/Build.gitlab-ci.yml and Jobs/Deploy/ECS.gitlab-ci.yml.

There are strict configuration rules regarding these templates:

• Only the main AWS/Deploy-ECS.gitlab-ci.yml template should be included.
• The individual Build and Deploy/ECS templates must not be included on their own, as they are designed to function only within the main template and may change unexpectedly.
• Job names within these templates should not be overridden in the pipeline configuration, as updates to the template may break the override functionality.

Additionally, users can control the deployment behavior using environment variables. Specifically, setting CI_AWS_ECS_WAIT_FOR_ROLLOUT_COMPLETE_DISABLED to a non-empty value will disable the behavior where the pipeline waits for the ECS rollout to complete.

Administrative Operations and Lifecycle Management

Maintaining the health and cost-efficiency of a GitLab EC2 deployment requires active administrative oversight.

Monitoring is achieved by connecting to the runner EC2 instances and examining the CloudFormation logs located at /var/log/cfn-*.log. These logs provide critical insights into why a stack may have failed to deploy or where configuration errors occurred during the initialization phase.

The lifecycle of the GitLab Runner architecture is managed through the CloudFormation stack. When the infrastructure is no longer needed, the most efficient method for cleaning up is the deletion of the entire CloudFormation stack. This action ensures that all provisioned resources—including EC2 instances, security groups, and network interfaces—are removed simultaneously, preventing "zombie" resources from incurring ongoing charges.

Analysis of Deployment Strategies

The transition from manual deployment to an IaC-driven approach on AWS provides a significant shift in operational stability. By utilizing a custom AMI and Launch Templates, organizations eliminate the "snowflake server" problem, where individual runners have slightly different configurations that lead to intermittent pipeline failures.

The use of the c5.2xlarge instance type highlights the resource-intensive nature of CI/CD processes. Since runners often perform compilation, testing, and containerization, the compute-optimized nature of the C5 family is essential for reducing the "time to feedback" for developers.

From a cost-optimization perspective, the integration of Auto Scaling Groups with GitLab Runners is the most impactful decision. Because CI/CD workloads are typically bursty, maintaining a fixed fleet of runners is wasteful. The ability to scale down to zero or a minimum baseline and scale up instantly upon the arrival of new jobs in the .gitlab-ci.yml queue ensures that performance is maintained without sacrificing the budget.

Furthermore, the reliance on S3 for artifacts and the use of the gl-ec2 script suite creates a decoupled deployment pipeline. By separating the build phase (S3 push) from the deploy phase (EC2 deployment), GitLab ensures that the deployment can be retried or rolled back without needing to re-run the entire build process, significantly increasing the agility of the release cycle.

Sources

1. Deploy and Manage Gitlab Runners on Amazon EC2
2. Install GitLab on AWS
3. Cloud Deployment
4. GitLab Cloud Deploy Merge Request 25