External object storage

GitLab relies on object storage for highly-available persistent data in Kubernetes. By default, an S3-compatible storage solution named minio is deployed with the chart, but for production quality deployments, we recommend using a hosted object storage solution like Google Cloud Storage or AWS S3.

To disable MinIO, set this option and then follow the related documentation below: 

--set global.minio.enabled=false

An example of the full configuration has been provided in the examples.

This documentation specifies usage of access and secret keys for AWS. It is also possible to use IAM roles.

Note: GitLab does not currently support using Amazon KMS to encrypt data stored in S3 buckets. Adding KMS support is being discussed in issue #1012.

Azure Blob Storage

GitLab uses fog, but doesn’t currently support Fog Azure. To make use Azure Blob Storage, you will have to set up an Azure MinIO gateway.

Docker Registry images

Configuration of object storage for the registry chart is done via the registry.storage key, and the global.registry.bucket key. 

--set registry.storage.secret=registry-storage
--set registry.storage.key=config
--set global.registry.bucket=bucket-name

Note: The bucket name needs to be set both in the secret, and in global.registry.bucket. The secret is used in the registry server, and the global is used by GitLab backups.

Create the secret per registry chart documentation on storage, then configure the chart to make use of this secret.

Examples for S3(any s3 compatible), Azure and GCS drivers can be found in examples/objectstorage.

Registry configuration

  1. Decide on which storage service to use.
  2. Copy appropriate file to registry-storage.yaml.
  3. Edit with the correct values for the environment.
  4. Follow registry chart documentation on storage for creating the secret.
  5. Configure the chart as documented.

LFS, Artifacts, Uploads, Packages, External Diffs, Pseudonymizer, Terraform State

Configuration of object storage for LFS, artifacts, uploads, packages, external diffs, and pseudonymizer is done via the global.appConfig.lfs, global.appConfig.artifacts, global.appConfig.uploads, global.appConfig.packages, global.appConfig.externalDiffs and global.appConfig.pseudonymizer keys. 

--set global.appConfig.lfs.bucket=gitlab-lfs-storage
--set global.appConfig.lfs.connection.secret=object-storage
--set global.appConfig.lfs.connection.key=connection

--set global.appConfig.artifacts.bucket=gitlab-artifacts-storage
--set global.appConfig.artifacts.connection.secret=object-storage
--set global.appConfig.artifacts.connection.key=connection

--set global.appConfig.uploads.bucket=gitlab-uploads-storage
--set global.appConfig.uploads.connection.secret=object-storage
--set global.appConfig.uploads.connection.key=connection

--set global.appConfig.packages.bucket=gitlab-packages-storage
--set global.appConfig.packages.connection.secret=object-storage
--set global.appConfig.packages.connection.key=connection

--set global.appConfig.externalDiffs.bucket=gitlab-externaldiffs-storage
--set global.appConfig.externalDiffs.connection.secret=object-storage
--set global.appConfig.externalDiffs.connection.key=connection

--set global.appConfig.terraformState.bucket=gitlab-terraform-state
--set global.appConfig.terraformState.connection.secret=object-storage
--set global.appConfig.terraformState.connection.key=connection

--set global.appConfig.pseudonymizer.bucket=gitlab-pseudonymizer-storage
--set global.appConfig.pseudonymizer.connection.secret=object-storage
--set global.appConfig.pseudonymizer.connection.key=connection
Note: Currently a different bucket is needed for each, otherwise performing a restore from backup will not properly function.
Note: Storing MR diffs on external storage is not enabled by default. So, for the object storage settings for externalDiffs to take effect, global.appConfig.externalDiffs.enabled key should have a true value.

See the charts/globals documentation on appConfig for full details.

Create the secret(s) per the connection details documentation, and then configure the chart to use the provided secrets. Note, the same secret can be used for all of them.

Examples for AWS (any S3 compatible like Azure using MinIO) and Google providers can be found in examples/objectstorage.

appConfig configuration

  1. Decide on which storage service to use.
  2. Copy appropriate file to rails.yaml.
  3. Edit with the correct values for the environment.
  4. Follow connection details documentation for creating the secret.
  5. Configure the chart as documented.

Backups

Backups are also stored in object storage, and need to be configured to point externally rather than the included MinIO service. The backup/restore procedure makes use of two separate buckets. A bucket for storing backups (global.appConfig.backups.bucket) and a tmp bucket for preserving existing data during the restore process (global.appConfig.backups.tmpBucket). Currently AWS S3-compatible object storage systems and Google Cloud Storage are supported backends The backend type is configurable by setting global.appConfig.backups.objectStorage.backend to s3 and gcs respectively. A connection configuration through the gitlab.task-runner.backups.objectStorage.config key must also be provided. When using Google Cloud Storage, the GCP project must be set with the global.appConfig.backups.objectStorage.config.gcpProject value.

For S3-compatible storage: 

--set global.appConfig.backups.bucket=gitlab-backup-storage
--set global.appConfig.backups.tmpBucket=gitlab-tmp-storage
--set gitlab.task-runner.backups.objectStorage.config.secret=storage-config
--set gitlab.task-runner.backups.objectStorage.config.key=config

For Google Cloud Storage (GCS): 

--set global.appConfig.backups.bucket=gitlab-backup-storage
--set global.appConfig.backups.tmpBucket=gitlab-tmp-storage
--set gitlab.task-runner.backups.objectStorage.backend=gcs
--set gitlab.task-runner.backups.objectStorage.config.gcpProject=my-gcp-project-id
--set gitlab.task-runner.backups.objectStorage.config.secret=storage-config
--set gitlab.task-runner.backups.objectStorage.config.key=config

See the backup/restore object storage documentation for full details.

Note: In order to backup/restore files from the other object storage locations, the config file needs to be configured to authenticate as a user with sufficient access to read/write to all GitLab buckets.

Backups storage example

  1. Create the storage.config file:

    • On Amazon S3, the contents should be in the s3cmd config file format 

      [default]
access_key = BOGUS_ACCESS_KEY
secret_key = BOGUS_SECRET_KEY
bucket_location = us-east-1

    • On Google Cloud Storage, you can create the file by creating a service account with the storage.admin role and then creating a service account key. Below is an example of using the gcloud CLI to create the file. 

      export PROJECT_ID=$(gcloud config get-value project)
gcloud iam service-accounts create gitlab-gcs --display-name "Gitlab Cloud Storage"
gcloud projects add-iam-policy-binding --role roles/storage.admin ${PROJECT_ID} --member=serviceAccount:gitlab-gcs@${PROJECT_ID}.iam.gserviceaccount.com
gcloud iam service-accounts keys create --iam-account gitlab-gcs@${PROJECT_ID}.iam.gserviceaccount.com storage.config

    • On Azure Storage 

      [default]
# Setup endpoint: hostname of the Web App
host_base = https://your_minio_setup.azurewebsites.net
host_bucket = https://your_minio_setup.azurewebsites.net
# Leave as default
bucket_location = us-west-1
use_https = True

# Setup access keys
# Access Key = Azure Storage Account name
access_key =  BOGUS_ACCOUNT_NAME
# Secret Key = Azure Storage Account Key
secret_key = BOGUS_KEY

# Use S3 v4 signature APIs
signature_v2 = False

  2. Create the secret 

    kubectl create secret generic storage-config --from-file=config=storage.config