Containerization simplifies IT operations and nearly all standard software can be deployed in a container environment. Until today, I have operated GitLab as an omnibus installation. While Omnibus already greatly simplifies the deployment experience, there still needs to be done and configured a lot to run GitLab. Before GitLab supported Let’s Encrypt that even meant taking care of certificate renewal and making sure certificate configurations don’t break during upgrades.

As I intend to run more software on the server running GitLab, I wanted to setup a reverse proxy that distributes traffic accordingly. With Traefik being a fresh and small cloud native edge router I decided to migrate the Omnibus GitLab installation to a Docker deployment behind the Traefik proxy.

Traefik and GitLab

Traefik will be the main entry point for all traffic and forward it to the configured backends. The following diagram depicts the targeted architecture:

Traefik, GitLab and any other app will be running as a Docker container on the host. Traefik will solely handle all incoming HTTP/S traffic and proxy that through an internal Docker network to the target containers.

Note that Traefik does not yet support arbitrary TCP connections so we need to expose the GitLab SSH port directly to the host (as of Traefik 2.0 there will be TCP support!). All other application ports are on an internal Docker network only accessible through Traefik.

Preparing the Migration

The migration from GitLab omnibus to a GitLab Docker container works by backing up the omnibus installation and restoring into the container. Thus, to prepare the migration, we have to make a backup of the existing GitLab instance. If you are starting from scratch with a GitLab deployment you can skip this section.

Upgrade to Latest GitLab Version

In order to migrate from an Omnibus GitLab installation to a Docker installation we can simply do a backup and restore procedure. To safely do this, the Omnibus GitLab version needs to be identical to the Docker GitLab version. So to start with the migration we will first update the existing GitLab Omnibus installation to the latest version:

# update GitLab on origin server to latest version
sudo apt-get update
sudo apt-get install gitlab-ce

Disable Backup

If you have configured scheduled backups of your GitLab instance make sure to disable the backup script for now. We don’t want any automation to interfere with our old GitLab instance. If you don’t remember at least check your existing cron jobs:

# check existing cron jobs
crontab -l
# if backup scripts are configured make sure to disable them
crontab -e

Backup GitLab

To perform a GitLab backup you can use the gitlab-rake command as described in the GitLab backup documentation. If you installed the omnibus package to default locations the command should be:

# perform backup on origin server
/opt/gitlab/bin/gitlab-rake gitlab:backup:create

Before you continue, please make sure that the backup was successful and the backup file really exists. Also, the backup above should be a full backup including all repositories. Check the backup log and backup file size (or even contents) to see if everything has been successfully backed up.

Stop the GitLab instance

Now that we have the backup we can turn off the running GitLab instance:

$ gitlab-ctl stop
ok: down: alertmanager: 454s, normally up
ok: down: gitaly: 454s, normally up
ok: down: gitlab-monitor: 453s, normally up
ok: down: gitlab-workhorse: 453s, normally up
ok: down: logrotate: 452s, normally up
ok: down: nginx: 4384223s, normally up
ok: down: node-exporter: 451s, normally up
ok: down: postgres-exporter: 451s, normally up
ok: down: postgresql: 451s, normally up
ok: down: prometheus: 450s, normally up
ok: down: redis: 450s, normally up
ok: down: redis-exporter: 449s, normally up
ok: down: sidekiq: 448s, normally up
ok: down: unicorn: 447s, normally up

To make sure we can’t start the GitLab instance by accident anymore we can add a syntax error to the gitlab.rb configuration file:

$ head /etc/gitlab/gitlab.rb -n 6
Attention:
This GitLab instance is disabled and this sentence renders an invalid config
so that this GitLab instance cannot be started by accident.

## Configuration options with # in front are not active and they were
## valid at install time. Updating the package does not update this file
...

Configure Docker Compose GitLab Deployment

To deploy the GitLab container we can use docker-compose to define all options of the resulting Docker container as declarative YAML file. Without further ado here is the docker-compose.yml file that spins up Gitlab and Traefik:

To configure Traefik we need to configure the referenced traefik.toml file so that Traefik properly registers to the HTTP/HTTPS endpoints and knows which domains to proxy:

Also note the [acme] section, which automagically enables Let’s Encrypt support for all the domains.

I like put the docker-compose.yml and traefik.toml under version control to have my service configuration backed up and thus do not want any sensitive information in there. Docker supports .env files to store credentials which can be used to pass sensitive information such as S3 access keys for backups to the GitLab container. The sensitive information can then be used in the docker-compose.yml as seen above.

$ cat .env
S3_ACCESS_KEY_ID_GITLAB_BACKUP=<ACCESS_KEY>
S3_SECRET_ACCESS_KEY_GITLAB_BACKUP=<SECRET_KEY>

Launch GitLab and Traefik

Now that we have configured the docker-compose.yml and traefik.toml to our needs, it’s time to spin up the services! To do so, we first need to create the web network that we reference in the docker compose file:

docker network create web

Now we can simply use docker-compose to launch the services:

docker-compose up -d traefik
docker-compose up -d gitlab

If things won’t work on the first try you should always recreate the container if you change any of the configuration files. You may do so with the docker-compose CLI:

docker-compose recreate traefik

Once Gitlab is up-and-running and accessible through Traefik, we can initiate the data migration as described in the next section.

Import Backup and Migrate the GitLab Instance Data

To start with the data import from the backup file, we need to first place the backup at the correct location. Notice the './data:/var/opt/gitlab' volume exposed to the GitLab Docker container. The volume ensures all data written to disk by GitLab is written to the host file system so that it is not lost during container recreation. GitLab also places its backup files in the data folder, namely at ./data/backups/. Move the backup archive file you created in the first step of this blog post right into that directory.

Once the backup file is in place, we can call the GitLab restore script. Now that GitLab is running in a Docker container we need to run all GitLab CLI commands through docker-compose exec. To perform the restore execute the following commands:

# launch an interactive shell inside the GitLab docker container
$ docker-compose exec gitlab /bin/bash
# start the restore procedure
:bash$> gitlab-rake gitlab:backup:restore

As soon as the restore procedure successfully completes, you should be able to login through the web interface using your former credentials. Verify that all data is there as expected.

Backup Cron Job

Automatically backing up the GitLab instance is very important to not loose data in case of trouble. In the docker-compose.yml we already included backup configuration for GitLab, so all we need to do now is to invoke the backup. As GitLab runs in the Docker container, we need to initiate the backup script through docker-compose exec. To automatically do this, you may setup a cron job on the host with crontab -e similar to the following:

# GitLab Backup [Docker]
0 2 */5 * * cd /opt/apps/gitlab/ && docker-compose exec gitlab gitlab-rake gitlab:backup:create CRON=1
0 1 * * * cd /opt/apps/gitlab/ && docker-compose exec gitlab gitlab-rake gitlab:backup:create CRON=1 SKIP=repositories,lfs,registry

This will run a short and small backup every night at 1am to backup the GitLab metadata. Then, every five days a full backup including all repositories, Docker registries and LFS is being run.

This backup strategy is totally sufficient for my use case, as repositories that have been worked on in the last 5 days are usually distributed across developers machines. In case of total loss, the GitLab metadata is important (which is backed up daily) and the last days of work can then simply be pushed again by developers.

Conclusion

Migrating GitLab Omnibus to a Docker containerized setup turned out to be a very smooth transition. In conjunction with Traefik this is a powerful setup that allows more services to be run on the same host.