GitLab Infrastructure Orchestration and Migration on RHEL-Based Ecosystems

GitLab serves as a cornerstone for modern software engineering, acting as an open-source, powerful, robust, and scalable collaboration platform. It is engineered to support every stage of the DevOps lifecycle, providing a cohesive environment where teams can plan development processes, write and verify code, package software, and execute releases through integrated continuous delivery features. Beyond simple version control, the platform incorporates sophisticated project management tools such as issue trackers with the ability to move issues between projects, comprehensive time tracking, advanced branching mechanisms, and file locking. For high-level project oversight, GitLab provides custom notifications, detailed project roadmaps, and burndown charts for both project and group milestones. While often compared to GitHub, GitLab distinguishes itself by offering a highly customizable self-managed experience, allowing organizations to maintain absolute control over their source code and deployment pipelines.

Foundational Installation and Dependency Management

The deployment of GitLab on RHEL-based systems, such as CentOS or Rocky Linux, requires a precise sequence of dependency installations to ensure the environment is prepared for the application's complex requirements. The process begins with the installation of essential system utilities.

For RHEL-based systems, the following command must be executed to install necessary curl and policy tools:

sudo yum install curl policycoreutils-python-utils

The inclusion of policycoreutils-python-utils is critical for managing SELinux policies, which often interact with GitLab's file system requirements. Following the initial dependencies, the Postfix service must be deployed. Postfix is the primary mail transfer agent used by GitLab to send notification emails, such as password resets, merge request alerts, and pipeline failure notifications.

The installation and activation of Postfix are handled via the following sequence:

sudo yum install postfix
sudo systemctl start postfix
sudo systemctl enable postfix
sudo systemctl status postfix

During the Postfix installation process, a configuration window typically appears. To ensure proper mail routing, the operator should select ‘Internet Site‘ and provide the server’s external DNS for the ‘mail name‘ field before confirming. Failure to correctly configure Postfix can result in a complete loss of communication between the GitLab instance and its users, rendering critical account recovery and notification features useless.

Deployment Methodologies: Package vs. Self-Compiled

There are two primary paths for installing GitLab on a Linux environment: the official Linux package (RPM/DEB) and the self-compiled source installation.

Linux Package Installation

The Linux package installation is the strongly recommended path for the vast majority of production environments. This method utilizes runit to manage GitLab processes, providing a significant reliability advantage. The primary benefit of this architecture is the ability to automatically restart any GitLab process that crashes. A specific real-world example of this utility is seen with the Sidekiq background worker. In heavily utilized GitLab instances, the memory usage of Sidekiq tends to grow over time. The Linux package installation solves this systemic issue by allowing Sidekiq to terminate gracefully when it exceeds specific memory thresholds, at which point runit immediately restarts the process, maintaining system stability without manual intervention.

Self-Compiled Installation

The self-compiled route is available for all tiers, including Free, Premium, and Ultimate, and is specifically designed for those requiring granular control over the source files. However, this method is noted to be labor-intensive and error-prone. While official guides exist for Debian and Ubuntu, those attempting this on RHEL/CentOS are explicitly advised to use the Linux packages instead.

For those who proceed with a self-compiled installation, the configuration process is exhaustive and requires strict adherence to permission structures to avoid system failure.

The initial configuration starts by navigating to the installation directory:

cd /home/git/gitlab

The operator must then initialize the configuration and secrets files using the provided examples:

sudo -u git -H cp config/gitlab.yml.example config/gitlab.yml
sudo -u git -H editor config/gitlab.yml
sudo -u git -H cp config/secrets.yml.example config/secrets.yml
sudo -u git -H chmod 0600 config/secrets.yml

The critical nature of file permissions cannot be overstated. Incorrect ownership or permissions on the log and temporary directories will prevent the application from booting. The following commands establish the necessary access rights:

sudo chown -R git log/
sudo chown -R git tmp/
sudo chmod -R u+rwX,go-w log/
sudo chmod -R u+rwX tmp/
sudo chmod -R u+rwX tmp/pids/
sudo chmod -R u+rwX tmp/sockets/

Furthermore, the public uploads directory must be secured to ensure only the GitLab user has access, as these files are served via gitlab-workhorse:

sudo -u git -H mkdir -p public/uploads/
sudo chmod 0700 public/uploads

Finally, permissions must be set for CI job logs, artifacts, and Pages:

sudo chmod -R u+rwX builds/
sudo chmod -R u+rwX shared/artifacts/
sudo chmod -R ug+rwX shared/pages/

Migration Strategies from CentOS 7.9 to Rocky Linux 9.4

Migrating a GitLab instance across different OS versions and distributions is a high-risk operation that requires strict version parity between the source and destination servers. A common pitfall is attempting to migrate from an older version on the source server to a newer version on the destination server; GitLab requires the same version/release to be installed on both sides before a restore can be performed.

Versioning Constraints and Upgrade Paths

In a scenario where a server is running CentOS 7.9 with GitLab 15.3 and moving to Rocky Linux 9.4, a direct jump to a version like 17.1 is impossible. If the source server (CentOS 7.9) only offers an upgrade path to 16.9.1, the migration cannot proceed until the versions match.

The successful migration path involves:
1. Upgrading the source server to the highest possible version available on the old OS. For example, upgrading from 15.3.4-ce.0.el7 to 17.3.1-ce.0.el7 on CentOS 7.9.
2. Installing the exact same version on the new server (e.g., 17.3.1-ce.0.el9 on Rocky Linux 9.4).
3. Executing the backup and restore process.

The Migration Execution Workflow

The following sequence is the authoritative method for migrating a GitLab instance:

  • Install the new server.
  • Install the identical version of GitLab on the new server.
  • Perform a full backup of the old server.
  • Copy the critical configuration files: /etc/gitlab/gitlab.rb and /etc/gitlab/gitlab-secrets.json from the old server to the new server.
  • Transfer the backup archive to /var/opt/gitlab/backups on the new server.
  • Run the initial reconfigure to initialize the environment.
  • Execute the restore command.
  • Run the reconfigure command a second time to finalize the settings.

A critical technical detail regarding the Redis database is that dump.rdb does not need to be copied manually via scp. Redis contains in-memory data used during runtime; the restore process handles the necessary data state, and the server can be used immediately after reconfiguration.

GitLab Runner Implementation and Architecture

The GitLab Runner is the agent that executes the CI/CD pipelines. It is available for multiple architectures, including x86_64, aarch64, and armhfp.

Installation and Package Management

For RHEL-based systems, the runner is installed using RPM packages. The process involves downloading both the helper images and the runner package itself.

The download commands are as follows:

curl -LJO "https://s3.dualstack.us-east-1.amazonaws.com/gitlab-runner-downloads/latest/rpm/gitlab-runner-helper-images.rpm"
curl -LJO "https://s3.dualstack.us-east-1.amazonaws.com/gitlab-runner-downloads/latest/rpm/gitlab-runner_${arch}.rpm"

For FIPS compliant environments on RHEL (currently only supporting x86_64), the helper images are included within a single package:

curl -LJO "https://s3.dualstack.us-east-1.amazonaws.com/gitlab-runner-downloads/latest/rpm/gitlab-runner_x86_64-fips.rpm"

The installation is performed using the dnf package manager:

dnf install -y gitlab-runner-helper-images.rpm gitlab-runner_<arch>.rpm

To upgrade the runner, the latest package is downloaded and the same installation command is run:

dnf install -y gitlab-runner-helper-images.rpm gitlab-runner_<arch>.rpm

Alternatively, users may choose to install the GitLab Runner using a binary file for environments where package managers are unavailable or restricted.

Advanced Geo-Replication and Secondary Site Configuration

GitLab Geo allows for the creation of a secondary site to provide read-only mirrors of the primary site, which reduces latency for geographically distributed teams and provides a robust disaster recovery mechanism.

Secondary Site Configuration Parameters

The configuration of a Geo secondary site requires specific entries in the gitlab.rb file. The site must be explicitly assigned the secondary role:

roles(['geo_secondary_role'])

A unique identifier for the site is required:

gitlab_rails['geo_node_name'] = 'location-2'

The external URL can be identical to the primary site if a unified URL setup is being utilized:

external_url 'https://gitlab.example.com'

Database and Network Integration

The secondary site must be linked to the primary site's PostgreSQL database. This requires the database password and replication password:

gitlab_rails['db_password'] = 'your_database_password_here'
postgresql['sql_user_password'] = 'md5_hash_of_your_database_password'
postgresql['sql_replication_password'] = 'md5_hash_of_your_replication_password'

Network settings must be strictly defined to allow communication:

postgresql['listen_address'] = '10.0.2.10'
postgresql['md5_auth_cidr_addresses'] = ['10.0.2.10/32']

Object Storage and Monitoring

For the secondary site to function, it must connect to a shared object store (such as AWS S3). The configuration is as follows:

gitlab_rails['object_store']['connection'] = { 'provider' => 'AWS', 'region' => 'us-east-1', 'aws_access_key_id' => 'your_access_key', 'aws_secret_access_key' => 'your_secret_key' }

Monitoring is optional but recommended for maintaining site health. The following settings configure Prometheus and node exporters:

node_exporter['listen_address'] = '0.0.0.0:9100'
gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229'
gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '10.0.0.0/8']
gitaly['configuration'] = { prometheus_listen_addr: '0.0.0.0:9236' }

Finally, to allow communication across the Geo sites, the ActionCable allowed origins must be specified:

gitlab_rails['action_cable_allowed_origins'] = ['https://secondary.example.com', 'https://primary.example.com']

Technical Specifications Summary

The following table summarizes the critical components and their roles within the GitLab ecosystem on RHEL/CentOS.

Component Primary Purpose Critical Configuration Requirement
Postfix Email Notifications DNS External Mail Name
Sidekiq Background Job Processing Memory limit monitoring (via runit)
GitLab Runner CI/CD Execution Architecture matching (x86_64/aarch64)
Gitaly Git Storage Handling Prometheus port 9236
PostgreSQL Data Persistence MD5 Hash passwords for replication
Redis In-memory Caching Not required for manual migration scp

Conclusion

The deployment and migration of GitLab on RHEL-based systems, specifically the transition from CentOS 7.9 to Rocky Linux 9.4, represent a complex exercise in version management and system administration. The absolute necessity of version parity between the source and destination servers is the most critical factor; any attempt to restore a backup to a version that does not match the backup's origin will result in failure. The transition from CentOS 7.9 to Rocky Linux 9.4 can be achieved by leveraging the latest available versions on the legacy OS before moving to the new environment, a process that can take significant time—up to 10 hours for full backup, restore, and runner association.

Furthermore, the choice between package installation and self-compilation highlights a fundamental trade-off between convenience and control. The Linux package's use of runit for process management provides a level of operational resilience that is difficult to replicate in a self-compiled environment, particularly regarding the memory-intensive nature of Sidekiq. For organizations scaling their operations via GitLab Geo, the complexity increases, requiring meticulous alignment of PostgreSQL replication and S3 object storage. Ultimately, the stability of a GitLab instance on CentOS or Rocky Linux depends on the strict adherence to directory permissions, accurate dependency installation, and a phased approach to software upgrades.

Sources

  1. GitLab Forum: Migrating GitLab to New Server
  2. TecMint: Install and Configure GitLab on CentOS 7
  3. GitLab Docs: Self-compiled Installation
  4. GitLab Docs: Install GitLab Runner on Linux Manually
  5. GitLab Docs: Setup Two Single Node Sites for Geo

Related Posts