• The global section
  • log_format examples (truncated)
    • runner
    • text
    • json
  • How check_interval works
• The [session_server] section
• The [[runners]] section
  • How clone_url works
• The EXECUTORS
• The SHELLS
• The [runners.docker] section
  • The [[runners.docker.services]] section
  • Volumes in the [runners.docker] section
    • Example 1: adding a data volume
    • Example 2: mount a host directory as a data volume
  • Using a private container registry
    • Support for GitLab integrated registry
    • Precedence of Docker authorization resolving
    • Restrict allowed_images to private registry
• The [runners.parallels] section
• The [runners.virtualbox] section
• The [runners.ssh] section
• The [runners.machine] section
  • The [[runners.machine.autoscaling]] sections
  • Periods syntax
• The [runners.custom] section
• The [runners.cache] section
  • The [runners.cache.s3] section
  • The [runners.cache.gcs] section
• The [runners.kubernetes] section
• Helper image
  • Overriding the helper image
• The [runners.custom_build_dir] section
• The [runners.referees] section
  • Using the Metrics Runner Referee
  • Configuring the Metrics Runner Referee for a Runner
• Note

Advanced configuration

GitLab Runner configuration uses the TOML format.

The file to be edited can be found in:

1. /etc/gitlab-runner/config.toml on *nix systems when GitLab Runner is executed as root (this is also path for service configuration)
2. ~/.gitlab-runner/config.toml on *nix systems when GitLab Runner is executed as non-root
3. ./config.toml on other systems

The global section

This defines global settings of GitLab Runner.

Configuration example:

concurrent = 4
log_level = "warning"

log_format examples (truncated)

runner

Runtime platform                                    arch=amd64 os=darwin pid=37300 revision=HEAD version=development version
Starting multi-runner from /etc/gitlab-runner/config.toml...  builds=0
WARNING: Running in user-mode.
WARNING: Use sudo for system-mode:
WARNING: $ sudo gitlab-runner...

Configuration loaded                                builds=0
listen_address not defined, metrics & debug endpoints disabled  builds=0
[session_server].listen_address not defined, session endpoints disabled  builds=0

text

INFO[0000] Runtime platform                              arch=amd64 os=darwin pid=37773 revision=HEAD version="development version"
INFO[0000] Starting multi-runner from /etc/gitlab-runner/config.toml...  builds=0
WARN[0000] Running in user-mode.
WARN[0000] Use sudo for system-mode:
WARN[0000] $ sudo gitlab-runner...
INFO[0000]
INFO[0000] Configuration loaded                          builds=0
INFO[0000] listen_address not defined, metrics & debug endpoints disabled  builds=0
INFO[0000] [session_server].listen_address not defined, session endpoints disabled  builds=0

json

{"arch":"amd64","level":"info","msg":"Runtime platform","os":"darwin","pid":38229,"revision":"HEAD","time":"2020-06-05T15:57:35+02:00","version":"development version"}
{"builds":0,"level":"info","msg":"Starting multi-runner from /etc/gitlab-runner/config.toml...","time":"2020-06-05T15:57:35+02:00"}
{"level":"warning","msg":"Running in user-mode.","time":"2020-06-05T15:57:35+02:00"}
{"level":"warning","msg":"Use sudo for system-mode:","time":"2020-06-05T15:57:35+02:00"}
{"level":"warning","msg":"$ sudo gitlab-runner...","time":"2020-06-05T15:57:35+02:00"}
{"level":"info","msg":"","time":"2020-06-05T15:57:35+02:00"}
{"builds":0,"level":"info","msg":"Configuration loaded","time":"2020-06-05T15:57:35+02:00"}
{"builds":0,"level":"info","msg":"listen_address not defined, metrics \u0026 debug endpoints disabled","time":"2020-06-05T15:57:35+02:00"}
{"builds":0,"level":"info","msg":"[session_server].listen_address not defined, session endpoints disabled","time":"2020-06-05T15:57:35+02:00"}

How check_interval works

If there is more than one [[runners]] section in config.toml (let’s call them workers), the interval between requests to GitLab are more frequent than one could expect. GitLab Runner contains a loop that constantly schedules a request that should be made for a worker against the GitLab instance it’s configured for.

GitLab Runner tries to ensure that subsequent requests for one worker will be done in the specified interval, so the value of check_interval is divided by the number of the [[runners]] sections. The loop will next iterate over all sections, schedule a request for each of them, and will sleep for the calculated amount of time. Things get interesting when the workers are tied to a different GitLab instance. Consider the following example.

If one would set check_interval = 10, and there were 2 workers in total (runner-1 and runner-2), a subsequent request would be made each 10 seconds. The loop would look like:

1. Get check_interval value (10s).
2. Get list of workers (runner-1, runner-2).
3. Calculate the sleep interval (10s / 2 = 5s).
4. Start an infinite loop:
   1. Request a job for runner-1.
   2. Sleep for 5s.
   3. Request a job for runner-2.
   4. Sleep for 5s.
   5. Repeat.

So, a request from the Runner’s process is made each 5s. If runner-1 and runner-2 are connected to the same GitLab instance, it means that the request to this GitLab instance will receive a new request from this Runner also each 5s. But as you can see, between the first request for runner-1 and second request for runner-1 there are two sleeps taking 5s, so finally it’s ~10s between subsequent requests for runner-1. The same goes for runner-2. If you define more workers, the sleep interval will be smaller, but a request for a worker will be repeated after all requests for the other workers + their sleeps are called.

The [session_server] section

The section [session_server] is a system runner level configuration, so it should be specified at the root level, not per executor i.e. it should be outside [[runners]] section. The session server allows the user to interact with jobs that the Runner is responsible for. A good example of this is the interactive web terminal.

Both listen_address and advertise_address should be provided in the form of host:port, where host may be an IP address (e.g., 127.0.0.1:8093) or a domain (e.g., my-runner.example.com:8093). The Runner will create a TLS certificate automatically to have a secure connection.

If you want to disable the session server, just delete the [session_server] section and terminal support will be disabled.

Example:

[session_server]
  listen_address = "[::]:8093" #  listen on all available interfaces on port 8093
  advertise_address = "runner-host-name.tld:8093"
  session_timeout = 1800

The [[runners]] section

This defines one runner entry.

Example:

[[runners]]
  name = "ruby-2.6-docker"
  url = "https://CI/"
  token = "TOKEN"
  limit = 0
  executor = "docker"
  builds_dir = ""
  shell = ""
  environment = ["ENV=value", "LC_ALL=en_US.UTF-8"]
  clone_url = "http://gitlab.example.local"

How clone_url works

In cases where the GitLab instance is exposed to a URL which can’t be used by the runner, a clone_url can be configured. For example; GitLab is exposed to https://gitlab.example.com, but the runner can’t reach that because of a firewall setup. If the runner can reach the node on 192.168.1.23, the clone_url should be set to http://192.168.1.23.

Only if the clone_url is set, the runner will construct a clone URL in the form of http://gitlab-ci-token:s3cr3tt0k3n@192.168.1.23/namespace/project.git.

The EXECUTORS

There are a couple of available executors currently.

The SHELLS

There are a couple of available shells that can be run on different platforms.

The [runners.docker] section

This defines the Docker Container parameters.

The [[runners.docker.services]] section

Specify additional services that should be run with the build. Please visit the Docker Registry for the list of available applications. Each service will be run in a separate container and linked to the build.

Example:

[runners.docker]
  host = ""
  hostname = ""
  tls_cert_path = "/Users/ayufan/.boot2docker/certs"
  image = "ruby:2.6"
  memory = "128m"
  memory_swap = "256m"
  memory_reservation = "64m"
  oom_kill_disable = false
  cpuset_cpus = "0,1"
  cpus = "2"
  dns = ["8.8.8.8"]
  dns_search = [""]
  privileged = false
  userns_mode = "host"
  cap_add = ["NET_ADMIN"]
  cap_drop = ["DAC_OVERRIDE"]
  devices = ["/dev/net/tun"]
  disable_cache = false
  wait_for_services_timeout = 30
  cache_dir = ""
  volumes = ["/data", "/home/project/cache"]
  extra_hosts = ["other-host:127.0.0.1"]
  shm_size = 300000
  volumes_from = ["storage_container:ro"]
  links = ["mysql_container:mysql"]
  allowed_images = ["ruby:*", "python:*", "php:*"]
  allowed_services = ["postgres:9", "redis:*", "mysql:*"]
  [[runners.docker.services]]
    name = "mysql"
    alias = "db"
  [[runners.docker.services]]
    name = "redis:2.8"
    alias = "cache"
  [[runners.docker.services]]
    name = "postgres:9"
    alias = "postgres-db"
  [runners.docker.sysctls]
    "net.ipv4.ip_forward" = "1"

Volumes in the [runners.docker] section

You can find the complete guide of Docker volume usage here.

Let’s use some examples to explain how it work (assuming you have a working runner).

Example 1: adding a data volume

A data volume is a specially-designated directory within one or more containers that bypasses the Union File System. Data volumes are designed to persist data, independent of the container’s life cycle.

[runners.docker]
  host = ""
  hostname = ""
  tls_cert_path = "/Users/ayufan/.boot2docker/certs"
  image = "ruby:2.6"
  privileged = false
  disable_cache = true
  volumes = ["/path/to/volume/in/container"]

This will create a new volume inside the container at /path/to/volume/in/container.

Example 2: mount a host directory as a data volume

In addition to creating a volume using a data volume, you can also mount a directory from your Docker daemon’s host into a container. It’s useful when you want to store directories outside the container.

[runners.docker]
  host = ""
  hostname = ""
  tls_cert_path = "/Users/ayufan/.boot2docker/certs"
  image = "ruby:2.6"
  privileged = false
  disable_cache = true
  volumes = ["/path/to/bind/from/host:/path/to/bind/in/container:rw"]

This will use /path/to/bind/from/host of the CI host inside the container at /path/to/bind/in/container.

Using a private container registry

Notes:

• This feature requires GitLab Runner 1.8 or higher
• For GitLab Runner versions >= 0.6, <1.8 there was a partial support for using private registries, which required manual configuration of credentials on runner’s host. We recommend to upgrade your Runner to at least version 1.8 if you want to use private registries.
• Using private registries with the if-not-present pull policy may introduce security implications. To fully understand how pull policies work, read the pull policies documentation.

If you want to use private registries as a source of images for your builds, you can set the authorization configuration in the DOCKER_AUTH_CONFIG variable. It can be set in both GitLab Variables section of a project and in the config.toml file.

For a detailed example, visit the Using Docker images documentation.

The steps performed by the Runner can be summed up to:

1. The registry name is found from the image name.
2. If the value is not empty, the executor will search for the authentication configuration for this registry.
3. Finally, if an authentication corresponding to the specified registry is found, subsequent pulls will make use of it.

Now that the Runner is set up to authenticate against your private registry, learn how to configure .gitlab-ci.yml in order to use that registry.

Support for GitLab integrated registry

Note: To work automatically with private/protected images from GitLab integrated registry it needs at least GitLab CE/EE 8.14 and GitLab Runner 1.8.

Starting with GitLab CE/EE 8.14, GitLab will send credentials for its integrated registry along with the build data. These credentials will be automatically added to registries authorization parameters list.

After this authorization against the registry will be proceed like for configuration added with DOCKER_AUTH_CONFIG variable.

Thanks to this, in your builds you can use any image from you GitLab integrated registry, even if the image is private/protected. To fully understand for which images the builds will have access, read the New CI build permissions model documentation.

Precedence of Docker authorization resolving

As described above, GitLab Runner can authorize Docker against a registry by using credentials sent in different way. To find a proper registry, the following precedence is taken into account:

1. Credentials configured with DOCKER_AUTH_CONFIG.
2. Credentials configured locally on Runner’s host with ~/.docker/config.json or ~/.dockercfg files (e.g., by running docker login on the host).
3. Credentials sent by default with job’s payload (e.g., credentials for integrated registry described above).

The first found credentials for the registry will be used. So for example, if you add some credentials for the integrated registry with the DOCKER_AUTH_CONFIG variable, then the default credentials will be overridden.

Restrict allowed_images to private registry

For certain setups you will restrict access of the build jobs to Docker images which comes from your private Docker registry. In that case set

[runners.docker]
  ...
  allowed_images = ["my.registry.tld:5000/*:*"]

The [runners.parallels] section

This defines the Parallels parameters.

Example:

[runners.parallels]
  base_name = "my-parallels-image"
  template_name = ""
  disable_snapshots = false

The [runners.virtualbox] section

This defines the VirtualBox parameters. This executor relies on vboxmanage as executable to control VirtualBox machines so you have to adjust your PATH environment variable on Windows hosts: PATH=%PATH%;C:\Program Files\Oracle\VirtualBox.

Example:

[runners.virtualbox]
  base_name = "my-virtualbox-image"
  base_snapshot = "my-image-snapshot"
  disable_snapshots = false

The [runners.ssh] section

This defines the SSH connection parameters.

Example:

[runners.ssh]
  host = "my-production-server"
  port = "22"
  user = "root"
  password = "production-server-password"
  identity_file = ""

The [runners.machine] section

Added in GitLab Runner v1.1.0.

This defines the Docker Machine based autoscaling feature. More details can be found in the separate runners autoscale documentation.

The [[runners.machine.autoscaling]] sections

Example:

[runners.machine]
  IdleCount = 5
  IdleTime = 600
  MaxBuilds = 100
  MachineName = "auto-scale-%s"
  MachineDriver = "digitalocean"
  MachineOptions = [
      "digitalocean-image=coreos-stable",
      "digitalocean-ssh-user=core",
      "digitalocean-access-token=DO_ACCESS_TOKEN",
      "digitalocean-region=nyc2",
      "digitalocean-size=4gb",
      "digitalocean-private-networking",
      "engine-registry-mirror=http://10.11.12.13:12345"
  ]
  [[runners.machine.autoscaling]]
    Periods = ["* * 9-17 * * mon-fri *"]
    IdleCount = 50
    IdleTime = 3600
    Timezone = "UTC"
  [[runners.machine.autoscaling]]
    Periods = ["* * * * * sat,sun *"]
    IdleCount = 5
    IdleTime = 60
    Timezone = "UTC"

Periods syntax

The Periods setting contains an array of string patterns of time periods represented in a cron-style format. The line contains following fields:

[second] [minute] [hour] [day of month] [month] [day of week] [year]

Like in the standard cron configuration file, the fields can contain single values, ranges, lists and asterisks. A detailed description of the syntax can be found here.

The [runners.custom] section

Define configuration for the custom executor.

The [runners.cache] section

This defines the distributed cache feature. More details can be found in the runners autoscale documentation.

Below is a table containing a summary of config.toml, cli options and ENV variables for register:

The [runners.cache.s3] section

Configure S3 storage for cache. This section contains settings related to S3, that previously were present globally in the [runners.cache] section.

Example:

[runners.cache]
  Type = "s3"
  Path = "path/to/prefix"
  Shared = false
  [runners.cache.s3]
    ServerAddress = "s3.amazonaws.com"
    AccessKey = "AWS_S3_ACCESS_KEY"
    SecretKey = "AWS_S3_SECRET_KEY"
    BucketName = "runners-cache"
    BucketLocation = "eu-west-1"
    Insecure = false

The [runners.cache.gcs] section

Configure native support for Google Cloud Storage. Read the Google Cloud Storage Authentication documentation to check where these values come from.

Examples:

Credentials configured directly in config.toml file:

[runners.cache]
  Type = "gcs"
  Path = "path/to/prefix"
  Shared = false
  [runners.cache.gcs]
    AccessID = "cache-access-account@test-project-123456.iam.gserviceaccount.com"
    PrivateKey = "-----BEGIN PRIVATE KEY-----\nXXXXXX\n-----END PRIVATE KEY-----\n"
    BucketName = "runners-cache"

Credentials in JSON file downloaded from GCP:

[runners.cache]
  Type = "gcs"
  Path = "path/to/prefix"
  Shared = false
  [runners.cache.gcs]
    CredentialsFile = "/etc/gitlab-runner/service-account.json"
    BucketName = "runners-cache"

The [runners.kubernetes] section

This defines the Kubernetes parameters. See Kubernetes executor for additional parameters.

Example:

[runners.kubernetes]
  host = "https://45.67.34.123:4892"
  cert_file = "/etc/ssl/kubernetes/api.crt"
  key_file = "/etc/ssl/kubernetes/api.key"
  ca_file = "/etc/ssl/kubernetes/ca.crt"
  image = "golang:1.8"
  privileged = true
  image_pull_secrets = ["docker-registry-credentials"]
  [runners.kubernetes.node_selector]
    gitlab = "true"

Helper image

When one of docker, docker+machine or kubernetes executors is used, GitLab Runner uses a specific container to handle Git, artifacts and cache operations. This container is created from a special image, named helper image.

The helper image is based on Alpine Linux and it’s provided for amd64 and arm architectures. It contains a gitlab-runner-helper binary which is a special compilation of GitLab Runner binary, that contains only a subset of available commands, as well as Git, Git LFS, SSL certificates store, and basic configuration of Alpine.

When GitLab Runner is installed from the DEB/RPM packages, both images (amd64 and arm based) are installed on the host. When the Runner prepares the environment for the job execution, if the image in specified version (based on Runner’s Git revision) is not found on Docker Engine, it is automatically loaded. It works like that for both docker and docker+machine executors.

Things work a little different for the kubernetes executor or when GitLab Runner is installed manually. For manual installations, the gitlab-runner-helper binary is not included and for the kubernetes executor, the API of Kubernetes doesn’t allow loading the gitlab-runner-helper image from a local archive. In both cases, GitLab Runner will download the helper image from Docker Hub, from GitLab’s official repository gitlab/gitlab-runner-helper by using the Runner’s revision and architecture for defining which tag should be downloaded.

Overriding the helper image

In some cases, you may need to override the helper image. There are many reasons for doing this:

1. To speed up jobs execution: In environments with slower internet connection, downloading over and over again the same image from Docker Hub may generate a significant increase of a job’s timings. Downloading the helper image from a local registry (where the exact copy of gitlab/gitlab-runner-helper:XYZ is stored) may speed things up.

2. Security concerns: Many people don’t like to download external dependencies that were not checked before. There might be a business rule to use only dependencies that were reviewed and stored in local repositories.

3. Build environments without internet access: In some cases, jobs are being executed in an environment which has a dedicated, closed network (this doesn’t apply to the kubernetes executor where the image still needs to be downloaded from an external registry that is available at least to the Kubernetes cluster).

4. Additional software: Some users may want to install some additional software to the helper image, like openssh to support submodules accessible via git+ssh instead of git+http.

In any of the cases described above, it’s possible to configure a custom image using the helper_image configuration field, that is available for the docker, docker+machine and kubernetes executors:

[[runners]]
  (...)
  executor = "docker"
  [runners.docker]
    (...)
    helper_image = "my.registry.local/gitlab/gitlab-runner-helper:tag"

Note that the version of the helper image should be considered as strictly coupled with the version of GitLab Runner. As it was described above, one of the main reasons of providing such images is that Runner is using the gitlab-runner-helper binary, and this binary is compiled from part of GitLab Runner sources which is using an internal API that is expected to be the same in both binaries.

The Runner by default references to a gitlab/gitlab-runner-helper:XYZ image, where XYZ is based on the Runner’s architecture and Git revision. Starting with GitLab Runner 11.3 it’s possible to define the version of used image automatically, by using one of the version variables:

[[runners]]
  (...)
  executor = "docker"
  [runners.docker]
    (...)
    helper_image = "my.registry.local/gitlab/gitlab-runner-helper:x86_64-${CI_RUNNER_REVISION}"

With that configuration, GitLab Runner will instruct the executor to use the image in version x86_64-${CI_RUNNER_REVISION}, which is based on its compilation data. After updating the Runner to a new version, this will ensure that the Runner will try to download the proper image. This of course means that the image should be uploaded to the registry before upgrading the Runner, otherwise the jobs will start failing with a “No such image” error.

In GitLab Runner 13.2 and later, the helper image is tagged by $CI_RUNNER_VERSION in addition to $CI_RUNNER_REVISION. Both tags are valid and point to the same image.

[[runners]]
  (...)
  executor = "docker"
  [runners.docker]
    (...)
    helper_image = "my.registry.local/gitlab/gitlab-runner-helper:x86_64-v${CI_RUNNER_VERSION}"

The [runners.custom_build_dir] section

This section defines custom build directories parameters.

Please notice, that the feature - if not configured explicitly - will be enabled by default for kubernetes, docker, docker-ssh, docker+machine and docker-ssh+machine executors. It will be disabled by default for all other executors.

This feature requires that GIT_CLONE_PATH is within a path defined within runners.builds_dir. For the ease of using builds_dir the $CI_BUILDS_DIR variable can be used.

The feature is by default enabled only for docker and kubernetes executors as they provide a good way to separate resources. This feature can be explicitly enabled for any executor, but special care should be taken when using with executors that share builds_dir and have concurrent > 1.

Example:

[runners.custom_build_dir]
  enabled = true

The [runners.referees] section

Use Runner Referees to pass extra job monitoring data to GitLab. Runner referees are special workers within the Runner manager that query and collect additional data related to a job and upload their results to GitLab as job artifacts.

Using the Metrics Runner Referee

If the machine/container that is running the job exposes Prometheus metrics that are gathered by a Prometheus server, GitLab Runner can query the Prometheus server for the entirety of the job duration. After the metrics are received, they are uploaded as a job artifact which can be used for analysis later.

Currently, only the docker-machine executor supports the referee.

Configuring the Metrics Runner Referee for a Runner

Define [runner.referees] and [runner.referees.metrics] in your config.toml file within a [[runner]] section and add the following fields:

Here is a complete configuration example for node_exporter metrics:

[[runners]]
  [runners.referees]
    [runners.referees.metrics]
      prometheus_address = "http://localhost:9090"
      query_interval = 10
      metric_queries = [
        "arp_entries:rate(node_arp_entries{{selector}}[{interval}])",
        "context_switches:rate(node_context_switches_total{{selector}}[{interval}])",
        "cpu_seconds:rate(node_cpu_seconds_total{{selector}}[{interval}])",
        "disk_read_bytes:rate(node_disk_read_bytes_total{{selector}}[{interval}])",
        "disk_written_bytes:rate(node_disk_written_bytes_total{{selector}}[{interval}])",
        "memory_bytes:rate(node_memory_MemTotal_bytes{{selector}}[{interval}])",
        "memory_swap_bytes:rate(node_memory_SwapTotal_bytes{{selector}}[{interval}])",
        "network_tcp_active_opens:rate(node_netstat_Tcp_ActiveOpens{{selector}}[{interval}])",
        "network_tcp_passive_opens:rate(node_netstat_Tcp_PassiveOpens{{selector}}[{interval}])",
        "network_receive_bytes:rate(node_network_receive_bytes_total{{selector}}[{interval}])",
        "network_receive_drops:rate(node_network_receive_drop_total{{selector}}[{interval}])",
        "network_receive_errors:rate(node_network_receive_errs_total{{selector}}[{interval}])",
        "network_receive_packets:rate(node_network_receive_packets_total{{selector}}[{interval}])",
        "network_transmit_bytes:rate(node_network_transmit_bytes_total{{selector}}[{interval}])",
        "network_transmit_drops:rate(node_network_transmit_drop_total{{selector}}[{interval}])",
        "network_transmit_errors:rate(node_network_transmit_errs_total{{selector}}[{interval}])",
        "network_transmit_packets:rate(node_network_transmit_packets_total{{selector}}[{interval}])"
      ]

Metrics queries are in canonical_name:query_string format. The query string supports two variables that are replaced during execution:

For example, a shared Runner environment using the docker-machine executor would have a {selector} similar to node=shared-runner-123.

Note

If you’d like to deploy to multiple servers using GitLab CI, you can create a single script that deploys to multiple servers or you can create many scripts. It depends on what you’d like to do.