- Metrics dashboard visibility
- Adding custom metrics
- Editing additional metrics from the dashboard
- Setting up alerts for Prometheus metrics
- Taking action on incidents
Monitor CI/CD Environment metrics
Once configured, GitLab will attempt to retrieve performance metrics for any environment which has had a successful deployment.
GitLab will automatically scan the Prometheus server for metrics from known servers like Kubernetes and NGINX, and attempt to identify individual environments. The supported metrics and scan process is detailed in our Prometheus Metrics Library documentation.
You can view the performance dashboard for an environment by clicking on the monitoring button.
Metrics dashboard visibility
GitLab 13.0から導入されました。
You can set the visibility of the metrics dashboard to Only Project Members or Everyone With Access. When set to Everyone with Access, the metrics dashboard is visible to authenticated and non-authenticated users.
Adding custom metrics
- Introduced in GitLab Premium 10.6.
- Moved to GitLab Core 12.10.
Custom metrics can be monitored by adding them on the monitoring dashboard page. Once saved, they will be displayed on the environment performance dashboard provided that either:
- A connected Kubernetes cluster with the environment scope of
*
is used and Prometheus installed on the cluster - Prometheus is manually configured.
A few fields are required:
- Name: Chart title
- Type: Type of metric. Metrics of the same type will be shown together.
- Query: Valid PromQL query.
- Y-axis label: Y axis title to display on the dashboard.
-
Unit label: Query units, for example
req / sec
. Shown next to the value.
Multiple metrics can be displayed on the same chart if the fields Name, Type, and Y-axis label match between metrics. For example, a metric with Name Requests Rate
, Type Business
, and Y-axis label rec / sec
would display on the same chart as a second metric with the same values. A Legend label is suggested if this feature is used.
Editing additional metrics from the dashboard
GitLab 12.9で導入されました。
You can edit existing additional custom metrics by clicking the More actions dropdown and selecting Edit metric.
Setting up alerts for Prometheus metrics
Managed Prometheus instances
Introduced in GitLab Ultimate 11.2 for custom metrics, and 11.3 for library metrics.
For managed Prometheus instances using auto configuration, alerts for metrics can be configured directly in the performance dashboard.
To set an alert:
- Click on the ellipsis icon in the top right corner of the metric you want to create the alert for.
- Choose Alerts
- Set threshold and operator.
- Click Add to save and activate the alert.
To remove the alert, click back on the alert icon for the desired metric, and click Delete.
External Prometheus instances
- Introduced in GitLab Ultimate 11.8.
- Moved to GitLab Core in 12.10.
For manually configured Prometheus servers, a notify endpoint is provided to use with Prometheus webhooks. If you have manual configuration enabled, an Alerts section is added to Settings > Integrations > Prometheus. This contains the URL and Authorization Key. The Reset Key button will invalidate the key and generate a new one.
To send GitLab alert notifications, copy the URL and Authorization Key into the webhook_configs
section of your Prometheus Alertmanager configuration:
receivers:
name: gitlab
webhook_configs:
- http_config:
bearer_token: 9e1cbfcd546896a9ea8be557caf13a76
send_resolved: true
url: http://192.168.178.31:3001/root/manual_prometheus/prometheus/alerts/notify.json
...
In order for GitLab to associate your alerts with an environment, you need to configure a gitlab_environment_name
label on the alerts you set up in Prometheus. The value of this should match the name of your Environment in GitLab.
Taking action on incidents
- Introduced in GitLab Ultimate 11.11.
- From GitLab Ultimate 12.5, when GitLab receives a recovery alert, it will automatically close the associated issue.
Alerts can be used to trigger actions, like opening an issue automatically (disabled by default since 13.1
). To configure the actions:
- Navigate to your project’s Settings > Operations > Incidents.
- Enable the option to create issues.
- Choose the issue template to create the issue from.
- Optionally, select whether to send an email notification to the developers of the project.
- 変更を保存するをクリックします。
Once enabled, an issue will be opened automatically when an alert is triggered which contains values extracted from alert’s payload:
- Issue author:
GitLab Alert Bot
- Issue title: Extract from
annotations/title
,annotations/summary
orlabels/alertname
- Alert
Summary
: A list of properties-
starts_at
: Alert start time viastartsAt
-
full_query
: Alert query extracted fromgeneratorURL
- Optional list of attached annotations extracted from
annotations/*
-
- Alert GFM: GitLab Flavored Markdown from
annotations/gitlab_incident_markdown
When GitLab receives a Recovery Alert, it will automatically close the associated issue. This action will be recorded as a system message on the issue indicating that it was closed automatically by the GitLab Alert bot.
To further customize the issue, you can add labels, mentions, or any other supported quick action in the selected issue template, which will apply to all incidents. To limit quick actions or other information to only specific types of alerts, use the annotations/gitlab_incident_markdown
field.
Since version 12.2, GitLab will tag each incident issue with the incident
label automatically. If the label does not yet exist, it will be created automatically as well.
If the metric exceeds the threshold of the alert for over 5 minutes, an email will be sent to all Maintainers and Owners of the project.