- Forked version of Docker machine
- Preparing the environment
- Configuring GitLab Runner
- Upgrading the Runner
Install and register GitLab Runner for autoscaling with Docker Machine
The auto scale feature was introduced in GitLab Runner 1.1.0.
For an overview of the autoscale architecture, take a look at the comprehensive documentation on autoscaling.
Forked version of Docker machine
Because docker-machine
is in maintenance
mode, GitLab is
providing it’s own fork of
docker-machine
,
which is based on the latest master
branch of docker-machine
with
some additional patches for the following bugs:
- Make DigitalOcean driver RateLimit aware
- Add backoff to Google driver operations check
- Add
--google-min-cpu-platform
option for machine creation - Use cached IP for Google driver
- Use cached IP for AWS driver
The intent of this fork is to fix critical and bugs affecting running costs only. No new features will be added.
Preparing the environment
In order to use the autoscale feature, Docker and GitLab Runner must be installed in the same machine:
- Log in to a new Linux-based machine that will serve as a bastion server where Docker will spawn new machines from
- Install GitLab Runner
- Install Docker Machine
- Optionally but recommended, prepare a proxy container registry and a cache server to be used with the autoscaled Runners
If you need to use any virtualization/cloud providers that aren’t handled by Docker’s Machine internal drivers, the appropriate driver plugin must be installed. The Docker Machine driver plugin installation and configuration is out of the scope of this documentation. For more details please read the Docker Machine documentation
Configuring GitLab Runner
-
Register a GitLab Runner and select the
docker+machine
executor when asked. - Edit
config.toml
and configure the Runner to use Docker machine. Visit the dedicated page covering detailed information about GitLab Runner Autoscaling. -
The first time you’re using Docker Machine, it’s best to execute manually
docker-machine create ...
with your chosen driver and all options from theMachineOptions
section. This will set up the Docker Machine environment properly and will also be a good validation of the specified options. After this, you can destroy the machine withdocker-machine rm [machine_name]
and start the Runner.Note: Multiple concurrent requests todocker-machine create
that are done at first usage are not good. When thedocker+machine
executor is used, the Runner may spin up few concurrentdocker-machine create
commands. If Docker Machine was not used before in this environment, each started process tries to prepare SSH keys and SSL certificates (for Docker API authentication between Runner and Docker Engine on the autoscaled spawned machine), and these concurrent processes are disturbing each other. This can end with a non-working environment. That’s why it’s important to create a test machine manually the very first time you set up the Runner with Docker Machine. - Now, you can try and start a new pipeline in your project. In a few seconds,
if you run
docker-machine ls
you should see a new machine being created.
Upgrading the Runner
- Check if your operating system is configured to automatically restart the
Runner (for example, by checking its service file):
-
if yes, ensure that service manager is configured to use
SIGQUIT
and use the service’s tools to stop the process:# For systemd sudo systemctl stop gitlab-runner # For upstart sudo service gitlab-runner stop
-
if no, you may stop the Runner’s process manually:
sudo killall -SIGQUIT gitlab-runner
Note: Sending theSIGQUIT
signal will make the Runner to stop gracefully. It will stop accepting new jobs, and will exit as soon as the current jobs are finished. -
-
Wait until the Runner exits. You can check its status with
gitlab-runner status
or await a graceful shutdown for up to 30 minutes with:for i in `seq 1 180`; do # 1800 seconds = 30 minutes gitlab-runner status || break sleep 10 done
- You can now safely install the new Runner without interrupting any jobs