- Installing applications
-
Install using GitLab CI/CD (alpha)
- 使用方法
- Important notes
- Install Ingress using GitLab CI/CD
- Install cert-manager using GitLab CI/CD
- Install Sentry using GitLab CI/CD
- Install PostHog using GitLab CI/CD
- Install Prometheus using GitLab CI/CD
- Install GitLab Runner using GitLab CI/CD
- Install Cilium using GitLab CI/CD
- Install Falco using GitLab CI/CD
- Install Vault using GitLab CI/CD
- Install JupyterHub using GitLab CI/CD
- Install Elastic Stack using GitLab CI/CD
- Install Crossplane using GitLab CI/CD
- Install Fluentd using GitLab CI/CD
- Install Knative using GitLab CI/CD
- Install AppArmor using GitLab CI/CD
- Upgrading applications
- Uninstalling applications
- Troubleshooting applications
GitLabマネージドアプリ
GitLab provides GitLab Managed Apps, a one-click install for various applications which can be added directly to your configured cluster.
These applications are needed for Review Apps and deployments when using Auto DevOps.
You can install them after you create a cluster.
Installing applications
Applications managed by GitLab will be installed onto the gitlab-managed-apps
namespace.
This namespace:
- Is different from the namespace used for project deployments.
- Is created once.
- Has a non-configurable name.
To see a list of available applications to install. For a:
- Project-level cluster, navigate to your project’s Operations > Kubernetes.
- Group-level cluster, navigate to your group’s Kubernetes page.
The following applications can be installed:
- Helm
- Ingress
- cert-manager
- Prometheus
- GitLab Runner
- JupyterHub
- Knative
- Crossplane
- Elastic Stack
- Fluentd
With the exception of Knative, the applications will be installed in a dedicated
namespace called gitlab-managed-apps
.
Helm
- Introduced in GitLab 10.2 for project-level clusters.
- Introduced in GitLab 11.6 for group-level clusters.
- A local Tiller option was introduced in GitLab 13.2 behind a feature flag, enabled by default.
- The feature flag for local Tiller is enabled on GitLab.com.
Helm is a package manager for Kubernetes and is
used to install the GitLab-managed apps. GitLab runs each helm
command
in a pod within the gitlab-managed-apps
namespace inside the cluster.
As of GitLab 13.2, the integration uses a local Tiller by default. When using a local Tiller, the Helm application does not need to be installed and will not be shown in the list of applications.
Enable or disable local Tiller
- Introduced in GitLab 13.2
- The option to disable local Tiller is planned for removal in GitLab 13.3
Local Tiller is under development, but is ready for production use. It is deployed behind a feature flag that is enabled by default. GitLab administrators with access to the GitLab Rails console can enable it for your instance.
有効にするには:
# Instance-wide
Feature.enable(:managed_apps_local_tiller)
無効化するには:
# Instance-wide
Feature.disable(:managed_apps_local_tiller)
cert-manager
Introduced in GitLab 11.6 for project- and group-level clusters.
cert-manager is a native Kubernetes certificate management controller that helps with issuing certificates. Installing cert-manager on your cluster will issue a certificate by Let’s Encrypt and ensure that certificates are valid and up-to-date.
The chart used to install this application depends on the version of GitLab used. In:
- GitLab 12.3 and newer, the jetstack/cert-manager
chart is used with a
values.yaml
file. - GitLab 12.2 and older, the stable/cert-manager chart was used.
If you have installed cert-manager prior to GitLab 12.3, Let’s Encrypt will block requests from older versions of cert-manager.
To resolve this:
- Uninstall cert-manager (consider backing up any additional configuration).
- Install cert-manager again.
GitLab Runner
- Introduced in GitLab 10.6 for project-level clusters.
- Introduced in GitLab 11.10 for group-level clusters.
GitLab Runner is the open source project that is used to run your jobs and send the results back to GitLab. It is used in conjunction with GitLab CI/CD, the open-source continuous integration service included with GitLab that coordinates the jobs.
If the project is on GitLab.com, shared Runners are available (the first 2000 minutes are free, you can buy more later) and you do not have to deploy one if they are enough for your needs. If a project-specific Runner is desired, or there are no shared Runners, it is easy to deploy one.
Note that the deployed Runner will be set as privileged, which means it will essentially have root access to the underlying machine. This is required to build Docker images, so it is the default. Make sure you read the security implications before deploying one.
runner/gitlab-runner
chart is used to install this application, using
a preconfigured values.yaml
file. Customizing the installation by modifying this file is not supported.Ingress
- Introduced in GitLab 10.2 for project-level clusters.
- Introduced in GitLab 11.6 for group-level clusters.
Ingress provides load balancing, SSL termination, and name-based virtual hosting out of the box. It acts as a web proxy for your applications and is useful if you want to use Auto DevOps or deploy your own web apps.
The Ingress Controller installed is Ingress-NGINX, which is supported by the Kubernetes community.
In order to publish your web application, you first need to find the endpoint which will be either an IP address or a hostname associated with your load balancer.
To install it, click on the Install button for Ingress. GitLab will attempt to determine the external endpoint and it should be available within a few minutes.
Determining the external endpoint automatically
Introduced in GitLab 10.6.
After you install Ingress, the external endpoint should be available within a few minutes.
KUBE_INGRESS_BASE_DOMAIN
environment variable.If the endpoint doesn’t appear and your cluster runs on Google Kubernetes Engine:
- Check your Kubernetes cluster on Google Kubernetes Engine to ensure there are no errors on its nodes.
- Ensure you have enough Quotas on Google Kubernetes Engine. For more information, see Resource Quotas.
- Check Google Cloud’s Status to ensure they are not having any disruptions.
Once installed, you may see a ?
for “Ingress IP Address” depending on the
cloud provider. For EKS specifically, this is because the ELB is created
with a DNS name, not an IP address. If GitLab is still unable to
determine the endpoint of your Ingress or Knative application, you can
determine it manually.
stable/nginx-ingress
chart is used to install this application with a
values.yaml
file.Determining the external endpoint manually
If the cluster is on GKE, click the Google Kubernetes Engine link in the
Advanced settings, or go directly to the
Google Kubernetes Engine dashboard
and select the proper project and cluster. Then click Connect and execute
the gcloud
command in a local terminal or using the Cloud Shell.
If the cluster is not on GKE, follow the specific instructions for your
Kubernetes provider to configure kubectl
with the right credentials.
The output of the following examples will show the external endpoint of your
cluster. This information can then be used to set up DNS entries and forwarding
rules that allow external access to your deployed applications.
If you installed Ingress via the Applications, run the following command:
kubectl get service --namespace=gitlab-managed-apps ingress-nginx-ingress-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
Some Kubernetes clusters return a hostname instead, like Amazon EKS. For these platforms, run:
kubectl get service --namespace=gitlab-managed-apps ingress-nginx-ingress-controller -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
For Istio/Knative, the command will be different:
kubectl get svc --namespace=istio-system istio-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip} '
Otherwise, you can list the IP addresses of all load balancers:
kubectl get svc --all-namespaces -o jsonpath='{range.items[?(@.status.loadBalancer.ingress)]}{.status.loadBalancer.ingress[*].ip} '
%
on some Kubernetes versions, do not include it.The Ingress is now available at this address and will route incoming requests to
the proper service based on the DNS name in the request. To support this, a
wildcard DNS CNAME record should be created for the desired domain name. For example,
*.myekscluster.com
would point to the Ingress hostname obtained earlier.
Using a static IP
By default, an ephemeral external IP address is associated to the cluster’s load balancer. If you associate the ephemeral IP with your DNS and the IP changes, your apps will not be able to be reached, and you’d have to change the DNS record again. In order to avoid that, you should change it into a static reserved IP.
Read how to promote an ephemeral external IP address in GKE.
Pointing your DNS at the external endpoint
Once you’ve set up the external endpoint, you should associate it with a wildcard DNS
record such as *.example.com.
in order to be able to reach your apps. If your external endpoint is an IP address,
use an A record. If your external endpoint is a hostname, use a CNAME record.
Web Application Firewall (ModSecurity)
GitLab 12.7で導入されました。
A Web Application Firewall (WAF) examines traffic being sent or received, and can block malicious traffic before it reaches your application. The benefits of a WAF are:
- Real-time security monitoring for your application
- Logging of all your HTTP traffic to the application
- Access control for your application
- Highly configurable logging and blocking rules
Out of the box, GitLab provides you with a WAF known as ModSecurity
.
ModSecurity is a toolkit for real-time web application monitoring, logging, and access control. With GitLab’s offering, the OWASP’s Core Rule Set, which provides generic attack detection capabilities, is automatically applied.
This feature:
- Runs in “Detection-only mode” unless configured otherwise.
-
Is viewable by checking your Ingress controller’s
modsec
log for rule violations. For example:kubectl logs -n gitlab-managed-apps $(kubectl get pod -n gitlab-managed-apps -l app=nginx-ingress,component=controller --no-headers=true -o custom-columns=:metadata.name) modsecurity-log -f
To enable WAF, switch its respective toggle to the enabled position when installing or updating Ingress application.
If this is your first time using GitLab’s WAF, we recommend you follow the quick start guide.
There is a small performance overhead by enabling ModSecurity. If this is considered significant for your application, you can disable ModSecurity’s rule engine for your deployed application in any of the following ways:
-
Setting the deployment variable
AUTO_DEVOPS_MODSECURITY_SEC_RULE_ENGINE
toOff
. This will prevent ModSecurity from processing any requests for the given application or environment. -
Switching its respective toggle to the disabled position and applying changes through the Save changes button. This will reinstall Ingress with the recent changes.
Logging and blocking modes
To help you tune your WAF rules, you can globally set your WAF to either Logging or Blocking mode:
- Logging mode - Allows traffic matching the rule to pass, and logs the event.
- Blocking mode - Prevents traffic matching the rule from passing, and logs the event.
To change your WAF’s mode:
- Install ModSecurity if you have not already done so.
- Navigate to Operations > Kubernetes.
- In Applications, scroll to Ingress.
- Under Global default, select your desired mode.
- 変更を保存するをクリックします。
WAF version updates
Enabling, disabling, or changing the logging mode for ModSecurity is only allowed within same version of Ingress due to limitations in Helm which might be overcome in future releases.
ModSecurity UI controls are disabled if the version deployed differs from the one available in GitLab, while actions at the Ingress level, such as uninstalling, can still be performed:
Updating Ingress to the most recent version enables you to take advantage of bug fixes, security fixes, and performance improvements. To update Ingress application, you must first uninstall it, and then re-install it as described in Install ModSecurity.
Viewing Web Application Firewall traffic
Introduced in GitLab Ultimate 12.9.
You can view Web Application Firewall traffic by navigating to your project’s Security & Compliance > Threat Monitoring page.
From there, you can see tracked over time:
- The total amount of traffic to your application.
- The proportion of traffic that is considered anomalous by the Web Application Firewall’s default OWASP ruleset.
If a significant percentage of traffic is anomalous, it should be investigated for potential threats, which can be done by examining the Web Application Firewall logs.
JupyterHub
- Introduced in GitLab 11.0 for project-level clusters.
- Introduced in GitLab 12.3 for group and instance-level clusters.
JupyterHub is a multi-user service for managing notebooks across a team. Jupyter Notebooks provide a web-based interactive programming environment used for data analysis, visualization, and machine learning.
Authentication will be enabled only for project members for project-level clusters and group members for group-level clusters with Developer or higher access to the associated project or group.
We use a custom Jupyter image that installs additional useful packages on top of the base Jupyter. You will also see ready-to-use DevOps Runbooks built with Nurtch’s Rubix library.
More information on creating executable runbooks can be found in our Runbooks documentation. Note that Ingress must be installed and have an IP address assigned before JupyterHub can be installed.
jupyter/jupyterhub
chart is used to install this application with a
values.yaml
file.Jupyter Git Integration
- Introduced in GitLab 12.0 for project-level clusters.
- Introduced in GitLab 12.3 for group and instance-level clusters.
When installing JupyterHub onto your Kubernetes cluster, JupyterLab’s Git extension is automatically provisioned and configured using the authenticated user’s:
- Name.
- Email.
- Newly created access token.
JupyterLab’s Git extension enables full version control of your notebooks as well as issuance of Git commands within Jupyter. Git commands can be issued via the Git tab on the left panel or via Jupyter’s command line prompt.
You can clone repositories from the files tab in Jupyter:
Knative
- Introduced in GitLab 11.5 for project-level clusters.
- Introduced in GitLab 12.3 for group- and instance-level clusters.
Knative provides a platform to create, deploy, and manage serverless workloads from a Kubernetes cluster. It is used in conjunction with, and includes Istio to provide an external IP address for all programs hosted by Knative.
You will be prompted to enter a wildcard
domain where your applications will be exposed. Configure your DNS
server to use the external IP address for that domain. For any
application created and installed, they will be accessible as
<program_name>.<kubernetes_namespace>.<domain_name>
. This will require
your Kubernetes cluster to have RBAC
enabled.
knative/knative
chart is used to install this application.Prometheus
- Introduced in GitLab 10.4 for project-level clusters.
- Introduced in GitLab 11.11 for group-level clusters.
Prometheus is an open-source monitoring and alerting system useful to supervise your deployed applications.
GitLab is able to monitor applications automatically, using the Prometheus integration. Kubernetes container CPU and memory metrics are automatically collected, and response metrics are retrieved from NGINX Ingress as well.
To enable monitoring, simply install Prometheus into the cluster with the Install button.
stable/prometheus
chart is used to install this application with a
values.yaml
file.Crossplane
- Introduced in GitLab 12.5 for project-level clusters.
Crossplane is a multi-cloud control plane useful for managing applications and infrastructure across multiple clouds. It extends the Kubernetes API using:
- Custom resources.
- Controllers that watch those custom resources.
Crossplane allows provisioning and lifecycle management of infrastructure components across cloud providers in a uniform manner by abstracting cloud provider-specific configurations.
The Crossplane GitLab-managed application:
- Installs Crossplane with a provider of choice on a Kubernetes cluster attached to the project repository.
- Can then be used to provision infrastructure or managed applications such as PostgreSQL (for example, CloudSQL from GCP or RDS from AWS) and other services required by the application via the Auto DevOps pipeline.
For information on configuring Crossplane installed on the cluster, see Crossplane configuration.
alpha/crossplane
chart v0.4.1 is used to
install Crossplane using the
values.yaml
file.Elastic Stack
Introduced in GitLab 12.7 for project- and group-level clusters.
Elastic Stack is a complete end-to-end log analysis solution which helps in deep searching, analyzing and visualizing the logs generated from different machines.
GitLab is able to gather logs from pods in your cluster automatically. Filebeat will run as a DaemonSet on each node in your cluster, and it will ship container logs to Elasticsearch for querying. GitLab will then connect to Elasticsearch for logs instead of the Kubernetes API, and you will have access to more advanced querying capabilities.
Log data is automatically deleted after 30 days using Curator.
To enable log shipping:
- Ensure your cluster contains at least 3 nodes of instance types larger than
f1-micro
,g1-small
, orn1-standard-1
. - Navigate to Operations > Kubernetes.
- In Kubernetes Cluster, select a cluster.
- In the Applications section, find Elastic Stack and click Install.
gitlab/elastic-stack
chart is used to install this application with a
values.yaml
file.f1-micro
, g1-small
, n1-standard-1
, or
*-highcpu-2
instance types.Optional: deploy Kibana to perform advanced queries
If you are an advanced user and have direct access to your Kubernetes cluster using kubectl
and helm
, you can deploy Kibana manually.
The following assumes that helm
has been initialized with helm init
.
Save the following to kibana.yml
:
elasticsearch:
enabled: false
filebeat:
enabled: false
kibana:
enabled: true
elasticsearchHosts: http://elastic-stack-elasticsearch-master.gitlab-managed-apps.svc.cluster.local:9200
Then install it on your cluster:
helm repo add gitlab https://charts.gitlab.io
helm install --name kibana gitlab/elastic-stack --values kibana.yml
To access Kibana, forward the port to your local machine:
kubectl port-forward svc/kibana-kibana 5601:5601
Then, you can visit Kibana at http://localhost:5601
.
Fluentd
Introduced in GitLab 12.10 for project- and group-level clusters.
Fluentd is an open source data collector, which enables you to unify the data collection and consumption to better use and understand your data. Fluentd sends logs in syslog format.
To enable Fluentd:
- Navigate to Operations > Kubernetes and click Applications. You will be prompted to enter a host, port and protocol where the WAF logs will be sent to via syslog.
- Provide the host domain name or URL in SIEM Hostname.
- Provide the host port number in SIEM Port.
- Select a SIEM Protocol.
- Select at least one of the available logs (such as WAF or Cilium).
- 変更を保存するをクリックします。
Future apps
Interested in contributing a new GitLab managed app? Visit the development guidelines page to get started.
Install using GitLab CI/CD (alpha)
GitLab 12.6 で導入されました。
This alternative method allows users to install GitLab-managed
applications using GitLab CI/CD. It also allows customization of the
install using Helm values.yaml
files.
Supported applications:
- Ingress
- cert-manager
- Sentry
- GitLab Runner
- Cilium
- Falco
- Vault
- JupyterHub
- Elastic Stack
- Crossplane
- Fluentd
- Knative
- PostHog
- Prometheus
使用方法
You can find and import all the files referenced below in the example cluster applications project.
To install applications using GitLab CI/CD:
- Connect the cluster to a cluster management project.
-
In that project, add a
.gitlab-ci.yml
file with the following content:include: - template: Managed-Cluster-Applications.gitlab-ci.yml
Note: The job provided by this template connects to the cluster using tools provided in a custom Docker image. It requires that you have a runner registered with the Docker, Kubernetes, or Docker Machine executor. -
Add a
.gitlab/managed-apps/config.yaml
file to define which applications you would like to install. Define theinstalled
key astrue
to install the application andfalse
to uninstall the application. For example, to install Ingress:ingress: installed: true
- Optionally, define
.gitlab/managed-apps/<application>/values.yaml
file to customize values for the installed application.
A GitLab CI/CD pipeline will then run on the master
branch to install the
applications you have configured. In case of pipeline failure, the
output of the Helm
Tiller binary
will be saved as a CI job artifact.
Important notes
以下に注意してください。
- We recommend using the cluster management project exclusively for managing deployments to a cluster. Do not add your application’s source code to such projects.
- When you set the value for
installed
key back tofalse
, the application will be unprovisioned from the cluster. - If you update
.gitlab/managed-apps/<application>/values.yaml
with new values, the application will be redeployed.
Install Ingress using GitLab CI/CD
To install Ingress, define the .gitlab/managed-apps/config.yaml
file
with:
ingress:
installed: true
Ingress will then be installed into the gitlab-managed-apps
namespace
of your cluster.
You can customize the installation of Ingress by defining a
.gitlab/managed-apps/ingress/values.yaml
file in your cluster
management project. Refer to the
chart
for the available configuration options.
Install cert-manager using GitLab CI/CD
cert-manager is installed using GitLab CI/CD by defining configuration in
.gitlab/managed-apps/config.yaml
.
cert-manager:
- Is installed into the
gitlab-managed-apps
namespace of your cluster. - Can be installed with or without a default Let’s Encrypt
ClusterIssuer
, which requires an email address to be specified. The email address is used by Let’s Encrypt to contact you about expiring certificates and issues related to your account.
The following configuration is required to install cert-manager using GitLab CI/CD:
certManager:
installed: true
letsEncryptClusterIssuer:
installed: true
email: "user@example.com"
The following installs cert-manager using GitLab CI/CD without the default ClusterIssuer
:
certManager:
installed: true
letsEncryptClusterIssuer:
installed: false
You can customize the installation of cert-manager by defining a
.gitlab/managed-apps/cert-manager/values.yaml
file in your cluster
management project. Refer to the
chart for the
available configuration options.
Install Sentry using GitLab CI/CD
To install Sentry, define the .gitlab/managed-apps/config.yaml
file
with:
sentry:
installed: true
Sentry will then be installed into the gitlab-managed-apps
namespace
of your cluster.
You can customize the installation of Sentry by defining
.gitlab/managed-apps/sentry/values.yaml
file in your cluster
management project. Refer to the
chart
for the available configuration options.
We recommend you pay close attention to the following configuration options:
-
email
. Needed to invite users to your Sentry instance and to send error emails. -
user
. Where you can set the login credentials for the default admin user. -
postgresql
. For a PostgreSQL password that can be used when running future updates.
postgresql.postgresqlPassword
key) or you will receive authentication errors. See the PostgreSQL chart documentation for more information.Here is an example configuration for Sentry:
# Admin user to create
user:
# Indicated to create the admin user or not,
# Default is true as the initial installation.
create: true
email: "<your email>"
password: "<your password>"
email:
from_address: "<your from email>"
host: smtp
port: 25
use_tls: false
user: "<your email username>"
password: "<your email password>"
enable_replies: false
ingress:
enabled: true
hostname: "<sentry.example.com>"
# Needs to be here between runs.
# See https://github.com/helm/charts/tree/master/stable/postgresql#upgrade for more info
postgresql:
postgresqlPassword: example-postgresql-password
Install PostHog using GitLab CI/CD
PostHog 🦔 is a developer-friendly, open-source product analytics platform.
To install PostHog into the gitlab-managed-apps
namespace of your cluster,
define the .gitlab/managed-apps/config.yaml
file with:
posthog:
installed: true
You can customize the installation of PostHog by defining .gitlab/managed-apps/posthog/values.yaml
in your cluster management project. Refer to the Configuration section of the PostHog chart’s README
for the available configuration options.
postgresql.postgresqlPassword
or you will receive authentication errors.
See the PostgreSQL chart documentation for more information.Redis pods are restarted between upgrades. To prevent downtime, provide a Redis
password using the redis.password
key. This prevents a new password from
being generated on each restart.
Here is an example configuration for PostHog:
ingress:
enabled: true
hostname: "<posthog.example.com>"
# This will be autogenerated if you skip it. Include if you have 2 or more web replicas
posthogSecret: 'long-secret-key-used-to-sign-cookies'
# Needs to be here between runs.
# See https://github.com/helm/charts/tree/master/stable/postgresql#upgrade for more info
postgresql:
postgresqlPassword: example-postgresql-password
# Recommended to set this to a value to redis prevent downtime between upgrades
redis:
password: example-redis-password
Install Prometheus using GitLab CI/CD
GitLab 12.8で導入されました。
Prometheus is an open-source monitoring and alerting system for supervising your deployed applications.
To install Prometheus into the gitlab-managed-apps
namespace of your cluster,
define the .gitlab/managed-apps/config.yaml
file with:
prometheus:
installed: true
You can customize the installation of Prometheus by defining
.gitlab/managed-apps/prometheus/values.yaml
in your cluster management
project. Refer to the
Configuration section of the Prometheus chart’s README
for the available configuration options.
Install GitLab Runner using GitLab CI/CD
GitLab Runner is installed using GitLab CI/CD by defining configuration in
.gitlab/managed-apps/config.yaml
.
The following configuration is required to install GitLab Runner using GitLab CI/CD:
gitlabRunner:
installed: true
GitLab Runner is installed into the gitlab-managed-apps
namespace of your cluster.
In order for GitLab Runner to function, you must specify the following:
-
gitlabUrl
- the GitLab server full URL (for example,https://example.gitlab.com
) to register the Runner against. -
runnerRegistrationToken
- The registration token for adding new Runners to GitLab. This must be retrieved from your GitLab instance.
These values can be specified using CI variables:
-
GITLAB_RUNNER_GITLAB_URL
will be used forgitlabUrl
. -
GITLAB_RUNNER_REGISTRATION_TOKEN
will be used forrunnerRegistrationToken
You can customize the installation of GitLab Runner by defining
.gitlab/managed-apps/gitlab-runner/values.yaml
file in your cluster
management project. Refer to the
chart for the
available configuration options.
Install Cilium using GitLab CI/CD
GitLab 12.8で導入されました。
Cilium is a networking plugin for Kubernetes that you can use to implement support for NetworkPolicy resources. For more information, see Network Policies.
For an overview, see the Container Network Security Demo for GitLab 12.8.
Enable Cilium in the .gitlab/managed-apps/config.yaml
file to install it:
# possible values are gke, eks or you can leave it blank
clusterType: gke
cilium:
installed: true
The clusterType
variable enables the recommended Helm variables for
a corresponding cluster type. The default value is blank. You can
check the recommended variables for each cluster type in the official
documentation:
You can customize Cilium’s Helm variables by defining the
.gitlab/managed-apps/cilium/values.yaml
file in your cluster
management project. Refer to the
Cilium chart
for the available configuration options.
By default, Cilium drops all disallowed packets upon policy
deployment. In
auditmode,
however, Cilium doesn’t drop disallowed packets. You can use
policy-verdict
log to observe policy-related decisions. You can
enable audit mode by adding the following to
.gitlab/managed-apps/cilium/values.yaml
:
config:
policyAuditMode: true
agent:
monitor:
eventTypes: ["drop", "policy-verdict"]
The Cilium monitor log for traffic is logged out by the
cilium-monitor
sidecar container. You can check these logs with the following command:
kubectl -n gitlab-managed-apps logs cilium-XXXX cilium-monitor
You can disable the monitor log in .gitlab/managed-apps/cilium/values.yaml
:
agent:
monitor:
enabled: false
The Hubble monitoring daemon is
enabled by default and it’s set to collect per namespace flow
metrics. This metrics are accessible on the Threat Monitoring
dashboard. You can disable Hubble by adding the following to
.gitlab/managed-apps/cilium/values.yaml
:
global:
hubble:
enabled: false
You can also adjust Helm values for Hubble via
.gitlab/managed-apps/cilium/values.yaml
:
global:
hubble:
enabled: true
metrics:
enabled:
- 'flow:sourceContext=namespace;destinationContext=namespace'
Install Falco using GitLab CI/CD
GitLab 13.1で導入されました。
GitLab Container Host Security Monitoring uses Falco as a runtime security tool that listens to the Linux kernel using eBPF. Falco parses system calls and asserts the stream against a configurable rules engine in real-time. For more information, see Falco’s Documentation.
You can enable Falco in the
.gitlab/managed-apps/config.yaml
file:
falco:
installed: true
You can customize Falco’s Helm variables by defining the
.gitlab/managed-apps/falco/values.yaml
file in your cluster
management project. Refer to the
Falco chart
for the available configuration options.
.gitlab/managed-apps/falco/values.yaml
:ebpf:
enabled: false
In rare cases where automatic probe installation on your cluster isn’t possible and the kernel/probe isn’t precompiled, you may need to manually prepare the kernel module or eBPF probe with driverkit and install it on each cluster node.
By default, Falco is deployed with a limited set of rules. To add more rules, add the following to
.gitlab/managed-apps/falco/values.yaml
(you can get examples from
Cloud Native Security Hub):
customRules:
file-integrity.yaml: |-
- rule: Detect New File
desc: detect new file created
condition: >
evt.type = chmod or evt.type = fchmod
output: >
File below a known directory opened for writing (user=%user.name
command=%proc.cmdline file=%fd.name parent=%proc.pname pcmdline=%proc.pcmdline gparent=%proc.aname[2])
priority: ERROR
tags: [filesystem]
- rule: Detect New Directory
desc: detect new directory created
condition: >
mkdir
output: >
File below a known directory opened for writing (user=%user.name
command=%proc.cmdline file=%fd.name parent=%proc.pname pcmdline=%proc.pcmdline gparent=%proc.aname[2])
priority: ERROR
tags: [filesystem]
By default, Falco only outputs security events to logs as JSON objects. To set it to output to an
external API
or application,
add the following to .gitlab/managed-apps/falco/values.yaml
:
falco:
programOutput:
enabled: true
keepAlive: false
program: mail -s "Falco Notification" someone@example.com
httpOutput:
enabled: true
url: http://some.url
You can check these logs with the following command:
kubectl logs -l app=falco -n gitlab-managed-apps
Install Vault using GitLab CI/CD
GitLab 12.9で導入されました。
Hashicorp Vault is a secrets management solution which can be used to safely manage and store passwords, credentials, certificates and more. A Vault installation could be leveraged to provide a single secure data store for credentials used in your applications, GitLab CI/CD jobs, and more. It could also serve as a way of providing SSL/TLS certificates to systems and deployments in your infrastructure. Leveraging Vault as a single source for all these credentials allows greater security by having a single source of access, control, and auditability around all your sensitive credentials and certificates.
To install Vault, enable it in the .gitlab/managed-apps/config.yaml
file:
vault:
installed: true
By default you will get a basic Vault setup with no scalable storage backend. This is enough for simple testing and small-scale deployments, though has limits to how much it can scale, and as it is a single instance deployment, you will experience downtime when upgrading the Vault application.
To optimally use Vault in a production environment, it’s ideal to have a good understanding
of the internals of Vault and how to configure it. This can be done by reading the
the Vault documentation as well as
the Vault Helm chart values.yaml
file.
At a minimum you will likely set up:
- A seal for extra encryption of the master key.
- A storage backend that is suitable for environment and storage security requirements.
- HA Mode.
- The Vault UI.
The following is an example values file (.gitlab/managed-apps/vault/values.yaml
)
that configures Google Key Management Service for auto-unseal, using a Google Cloud Storage backend, enabling
the Vault UI, and enabling HA with 3 pod replicas. The storage
and seal
stanzas
below are examples and should be replaced with settings specific to your environment.
# Enable the Vault WebUI
ui:
enabled: true
server:
# Disable the built in data storage volume as it's not safe for Hight Availability mode
dataStorage:
enabled: false
# Enable High Availability Mode
ha:
enabled: true
# Configure Vault to listen on port 8200 for normal traffic and port 8201 for inter-cluster traffic
config: |
listener "tcp" {
tls_disable = 1
address = "[::]:8200"
cluster_address = "[::]:8201"
}
# Configure Vault to store its data in a GCS Bucket backend
storage "gcs" {
path = "gcs://my-vault-storage/vault-bucket"
ha_enabled = "true"
}
# Configure Vault to automatically unseal storage using a GKMS key
seal "gcpckms" {
project = "vault-helm-dev-246514"
region = "global"
key_ring = "vault-helm-unseal-kr"
crypto_key = "vault-helm-unseal-key"
}
Once you have successfully installed Vault, you will need to initialize the Vault
and obtain the initial root token. You will need access to your Kubernetes cluster that Vault has been deployed into in order to do this.
To initialise the Vault, get a shell to one of the Vault pods running inside Kubernetes (typically this is done by using the kubectl
command line tool).
Once you have a shell into the pod, run the vault operator init
command:
kubectl -n gitlab-managed-apps exec -it vault-0 sh
/ $ vault operator init
This should give you your unseal keys and initial root token. Make sure to note these down and keep these safe as you will need them to unseal the Vault throughout its lifecycle.
Install JupyterHub using GitLab CI/CD
GitLab 12.8で導入されました。
JupyterHub is installed using GitLab CI/CD by defining configuration in
.gitlab/managed-apps/config.yaml
as follows:
jupyterhub:
installed: true
gitlabProjectIdWhitelist: []
gitlabGroupWhitelist: []
In the configuration:
-
gitlabProjectIdWhitelist
restricts GitLab authentication to only members of the specified projects. -
gitlabGroupWhitelist
restricts GitLab authentication to only members of the specified groups. - Specifying an empty array for both will allow any user on the GitLab instance to sign in.
JupyterHub is installed into the gitlab-managed-apps
namespace of your cluster.
For JupyterHub to function, you must set up an OAuth Application. Set:
- “Redirect URI” to
http://<JupyterHub Host>/hub/oauth_callback
. - “Scope” to
api read_repository write_repository
.
In addition, the following variables must be specified using CI variables:
CI Variable | 説明 |
---|---|
JUPYTERHUB_PROXY_SECRET_TOKEN
| Secure string used for signing communications from the hub. Seeproxy.secretToken .
|
JUPYTERHUB_COOKIE_SECRET
| Secure string used for signing secure cookies. See hub.cookieSecret .
|
JUPYTERHUB_HOST
| Hostname used for the installation. For example, jupyter.gitlab.example.com .
|
JUPYTERHUB_GITLAB_HOST
| Hostname of the GitLab instance used for authentication. For example, gitlab.example.com .
|
JUPYTERHUB_AUTH_CRYPTO_KEY
| A 32-byte encryption key used to set auth.state.cryptoKey .
|
JUPYTERHUB_AUTH_GITLAB_CLIENT_ID
| “Application ID” for the OAuth Application. |
JUPYTERHUB_AUTH_GITLAB_CLIENT_SECRET
| “Secret” for the OAuth Application. |
By default, JupyterHub will be installed using a
default values file.
You can customize the installation of JupyterHub by defining a
.gitlab/managed-apps/jupyterhub/values.yaml
file in your cluster management project.
Refer to the chart reference for the available configuration options.
Install Elastic Stack using GitLab CI/CD
GitLab 12.8で導入されました。
Elastic Stack is installed using GitLab CI/CD by defining configuration in
.gitlab/managed-apps/config.yaml
.
The following configuration is required to install Elastic Stack using GitLab CI/CD:
elasticStack:
installed: true
Elastic Stack is installed into the gitlab-managed-apps
namespace of your cluster.
You can check the default values.yaml
we set for this chart.
You can customize the installation of Elastic Stack by defining
.gitlab/managed-apps/elastic-stack/values.yaml
file in your cluster
management project. Refer to the
chart for the
available configuration options.
Install Crossplane using GitLab CI/CD
GitLab 12.9で導入されました。
Crossplane is installed using GitLab CI/CD by defining configuration in
.gitlab/managed-apps/config.yaml
.
The following configuration is required to install Crossplane using GitLab CI/CD:
Crossplane:
installed: true
Crossplane is installed into the gitlab-managed-apps
namespace of your cluster.
You can check the default
values.yaml
we set for this chart.
You can customize the installation of Crossplane by defining
.gitlab/managed-apps/crossplane/values.yaml
file in your cluster
management project. Refer to the
chart for the
available configuration options. Note that this link points to the documentation for the current development release, which may differ from the version you have installed.
Install Fluentd using GitLab CI/CD
Introduced in GitLab 12.10.
To install Fluentd into the gitlab-managed-apps
namespace of your cluster using GitLab CI/CD, define the following configuration in .gitlab/managed-apps/config.yaml
:
Fluentd:
installed: true
You can also review the default values set for this chart in the values.yaml
file.
You can customize the installation of Fluentd by defining
.gitlab/managed-apps/fluentd/values.yaml
file in your cluster management
project. Refer to the
configuration chart for the current development release of Fluentd
for the available configuration options.
Install Knative using GitLab CI/CD
To install Knative, define the .gitlab/managed-apps/config.yaml
file
with:
knative:
installed: true
You can customize the installation of Knative by defining .gitlab/managed-apps/knative/values.yaml
file in your cluster management project. Refer to the chart
for the available configuration options.
Here is an example configuration for Knative:
domain: 'my.wildcard.A.record.dns'
If you plan to use GitLab Serverless capabilities, be sure to set an A record wildcard domain on your custom configuration.
Knative Metrics
GitLab provides Invocation Metrics for your functions. To collect these metrics, you must have:
- Knative and Prometheus managed applications installed on your cluster.
-
Manually applied the custom metrics on your cluster by running the following command:
kubectl apply -f https://gitlab.com/gitlab-org/cluster-integration/cluster-applications/-/raw/02c8231e30ef5b6725e6ba368bc63863ceb3c07d/src/default-data/knative/istio-metrics.yaml
Uninstall Knative
To uninstall Knative, you must first manually remove any custom metrics you have added by running the following command:
kubectl delete -f https://gitlab.com/gitlab-org/cluster-integration/cluster-applications/-/raw/02c8231e30ef5b6725e6ba368bc63863ceb3c07d/src/default-data/knative/istio-metrics.yaml
Install AppArmor using GitLab CI/CD
GitLab 13.1で導入されました。
To install AppArmor into the gitlab-managed-apps
namespace of your cluster using GitLab CI/CD, define the following configuration in .gitlab/managed-apps/config.yaml
:
apparmor:
installed: true
You can define one or more AppArmor profiles by adding them into .gitlab/managed-apps/apparmor/values.yaml
as the following:
profiles:
profile-one: |-
profile profile-one {
file,
}
Refer to the AppArmor chart for more information on this chart.
Using AppArmor profiles in your deployments
After installing AppAmor, you can use profiles by adding Pod Annotations. If you’re using Auto
DevOps, you can customize auto-deploy-values.yaml
to annotate your pods. Although it’s helpful to be aware of the list of custom attributes, you’re only required to set
podAnnotations
as follows:
podAnnotations:
container.apparmor.security.beta.kubernetes.io/auto-deploy-app: localhost/profile-one
The only information to be changed here is the profile name which is profile-one
in this example. Refer to the AppArmor tutorial for more information on how AppArmor is integrated in Kubernetes.
Upgrading applications
Introduced in GitLab 11.8.
The applications below can be upgraded.
Application | GitLabバージョン |
---|---|
Runner | 11.8+ |
To upgrade an application:
- For a:
- Project-level cluster, navigate to your project’s Operations > Kubernetes.
- Group-level cluster, navigate to your group’s Kubernetes page.
- Select your cluster.
- If an upgrade is available, the Upgrade button is displayed. Click the button to upgrade.
runner
chart plus the values set by
values.yaml
Uninstalling applications
GitLab 11.11で導入されました。
The applications below can be uninstalled.
Application | GitLabバージョン | 備考 |
---|---|---|
cert-manager | 12.2+ | The associated private key will be deleted and cannot be restored. Deployed applications will continue to use HTTPS, but certificates will not be renewed. Before uninstalling, you may wish to back up your configuration or revoke your certificates. |
GitLab Runner | 12.2+ | Any running pipelines will be canceled. |
Helm | 12.2+ | The associated Tiller pod, the gitlab-managed-apps namespace, and all of its resources will be deleted and cannot be restored.
|
Ingress | 12.1+ | The associated load balancer and IP will be deleted and cannot be restored. Furthermore, it can only be uninstalled if JupyterHub is not installed. |
JupyterHub | 12.1+ | All data not committed to GitLab will be deleted and cannot be restored. |
Knative | 12.1+ | The associated IP will be deleted and cannot be restored. |
Prometheus | 11.11+ | All data will be deleted and cannot be restored. |
Crossplane | 12.5+ | All data will be deleted and cannot be restored. |
Elastic Stack | 12.7+ | All data will be deleted and cannot be restored. |
Sentry | 12.6+ | The PostgreSQL persistent volume will remain and should be manually removed for complete uninstall. |
To uninstall an application:
- For a:
- Project-level cluster, navigate to your project’s Operations > Kubernetes.
- Group-level cluster, navigate to your group’s Kubernetes page.
- Select your cluster.
- Click the Uninstall button for the application.
Support for uninstalling all applications is planned for progressive rollout. To follow progress, see the relevant epic.
Troubleshooting applications
Applications can fail with the following error:
Error: remote error: tls: bad certificate
To avoid installation errors:
- Before starting the installation of applications, make sure that time is synchronized between your GitLab server and your Kubernetes cluster.
-
Ensure certificates are not out of sync. When installing applications, GitLab expects a new cluster with no previous installation of Helm.
You can confirm that the certificates match via
kubectl
:kubectl get configmaps/values-content-configuration-ingress -n gitlab-managed-apps -o \ "jsonpath={.data['cert\.pem']}" | base64 -d > a.pem kubectl get secrets/tiller-secret -n gitlab-managed-apps -o "jsonpath={.data['ca\.crt']}" | base64 -d > b.pem diff a.pem b.pem
Error installing managed apps on EKS cluster
If you’re using a managed cluster on AWS EKS, and you are not able to install some of the managed apps, consider checking the logs.
You can check the logs by running following commands:
kubectl get pods --all-namespaces
kubectl get services --all-namespaces
If you are getting the Failed to assign an IP address to container
error, it’s probably due to the
instance type you’ve specified in the AWS configuration.
The number and size of nodes might not have enough IP addresses to run or install those pods.
For reference, all the AWS instance IP limits are found
in this AWS repository on GitHub (search for InstanceENIsAvailable
).