GitLab Developers Guide to Working with Gitaly

Gitaly is a high-level Git RPC service used by GitLab CE/EE, Workhorse and GitLab-Shell.

Deep Dive

In May 2019, Bob Van Landuyt hosted a Deep Dive (GitLab team members only: https://gitlab.com/gitlab-org/create-stage/issues/1) on GitLab’s Gitaly project and how to contribute to it as a Ruby developer, to share his domain specific knowledge with anyone who may work in this part of the code base in the future.

You can find the recording on YouTube, and the slides on Google Slides and in PDF.

Everything covered in this deep dive was accurate as of GitLab 11.11, and while specific details may have changed since then, it should still serve as a good introduction.

Beginner’s guide

Start by reading the Gitaly repository’s Beginner’s guide to Gitaly contributions. It describes how to set up Gitaly, the various components of Gitaly and what they do, and how to run its test suites.

Developing new Git features

To read or write Git data, a request has to be made to Gitaly. This means that if you’re developing a new feature where you need data that’s not yet available in lib/gitlab/git changes have to be made to Gitaly.

This is a new process that is not clearly defined yet. If you want to contribute a Git feature and you’re getting stuck, reach out to the Gitaly team or @jacobvosmaer-gitlab.

By ‘new feature’ we mean any method or class in lib/gitlab/git that is called from outside lib/gitlab/git. For new methods that are called from inside lib/gitlab/git, see ‘Modifying existing Git features’ below.

There should be no new code that touches Git repositories via disk access (e.g. Rugged, git, rm -rf) anywhere outside lib/gitlab/git.

The process for adding new Gitaly features is:

  • exploration / prototyping
  • design and create a new Gitaly RPC in gitaly-proto
  • release a new version of gitaly-proto
  • write implementation and tests for the RPC in Gitaly, in Go or Ruby
  • release a new version of Gitaly
  • write client code in GitLab CE/EE, GitLab Workhorse or GitLab Shell that calls the new Gitaly RPC

These steps often overlap. It is possible to use an unreleased version of Gitaly and gitaly-proto during testing and development.

  • See the Gitaly repository for instructions on writing server side code with an unreleased protocol.
  • See below for instructions on running GitLab CE tests with a modified version of Gitaly.
  • In GDK run gdk install and restart gdk run (or gdk run app) to use a locally modified Gitaly version for development

gitaly-ruby

It is possible to implement and test RPC’s in Gitaly using Ruby code, in gitaly-ruby. This should make it easier to contribute for developers who are less comfortable writing Go code.

There is documentation for this approach in the Gitaly repository.

If your test-suite is failing with Gitaly issues, as a first step, try running: 

rm -rf tmp/tests/gitaly

During RSpec tests, the Gitaly instance will write logs to gitlab/log/gitaly-test.log.

Legacy Rugged code

While Gitaly can handle all Git access, many of GitLab customers still run Gitaly atop NFS. The legacy Rugged implementation for Git calls may be faster than the Gitaly RPC due to N+1 Gitaly calls and other reasons. See the issue for more details.

Until GitLab has eliminated most of these inefficiencies or the use of NFS is discontinued for Git data, Rugged implementations of some of the most commonly-used RPCs can be enabled via feature flags:

  • rugged_find_commit
  • rugged_get_tree_entries
  • rugged_tree_entry
  • rugged_commit_is_ancestor
  • rugged_commit_tree_entry
  • rugged_list_commits_by_oid

A convenience Rake task can be used to enable or disable these flags all together. To enable: 

bundle exec rake gitlab:features:enable_rugged

To disable: 

bundle exec rake gitlab:features:disable_rugged

Most of this code exists in the lib/gitlab/git/rugged_impl directory.

Note: You should NOT need to add or modify code related to Rugged unless explicitly discussed with the Gitaly Team. This code will NOT work on GitLab.com or other GitLab instances that do not use NFS.

TooManyInvocationsError errors

During development and testing, you may experience Gitlab::GitalyClient::TooManyInvocationsError failures. The GitalyClient will attempt to block against potential n+1 issues by raising this error when Gitaly is called more than 30 times in a single Rails request or Sidekiq execution.

As a temporary measure, export GITALY_DISABLE_REQUEST_LIMITS=1 to suppress the error. This will disable the n+1 detection in your development environment.

Please raise an issue in the GitLab CE or EE repositories to report the issue. Include the labels ~Gitaly ~performance ~”technical debt”. Please ensure that the issue contains the full stack trace and error message of the TooManyInvocationsError. Also include any known failing tests if possible.

Isolate the source of the n+1 problem. This will normally be a loop that results in Gitaly being called for each element in an array. If you are unable to isolate the problem, please contact a member of the Gitaly Team for assistance.

Once the source has been found, wrap it in an allow_n_plus_1_calls block, as follows: 

# n+1: link to n+1 issue
Gitlab::GitalyClient.allow_n_plus_1_calls do
  # original code
  commits.each { |commit| ... }
end

Once the code is wrapped in this block, this code-path will be excluded from n+1 detection.

Request counts

Commits and other Git data, is now fetched through Gitaly. These fetches can, much like with a database, be batched. This improves performance for the client and for Gitaly itself and therefore for the users too. To keep performance stable and guard performance regressions, Gitaly calls can be counted and the call count can be tested against. This requires the :request_store flag to be set. 

describe 'Gitaly Request count tests' do
  context 'when the request store is activated', :request_store do
    it 'correctly counts the gitaly requests made' do
      expect { subject }.to change { Gitlab::GitalyClient.get_request_count }.by(10)
    end
  end
end

Running tests with a locally modified version of Gitaly

Normally, GitLab CE/EE tests use a local clone of Gitaly in tmp/tests/gitaly pinned at the version specified in GITALY_SERVER_VERSION. The GITALY_SERVER_VERSION file supports also branches and SHA to use a custom commit in https://gitlab.com/gitlab-org/gitaly.

Note: With the introduction of auto-deploy for Gitaly, the format of GITALY_SERVER_VERSION was aligned with Omnibus syntax. It no longer supports =revision, it will evaluate the file content as a Git reference (branch or SHA), only if it matches a semver it will prepend a v.

If you want to run tests locally against a modified version of Gitaly you can replace tmp/tests/gitaly with a symlink. This is much faster because if will avoid a Gitaly re-install each time you run rspec. 

rm -rf tmp/tests/gitaly
ln -s /path/to/gitaly tmp/tests/gitaly

Make sure you run make in your local Gitaly directory before running tests. Otherwise, Gitaly will fail to boot.

If you make changes to your local Gitaly in between test runs you need to manually run make again.

Note that CI tests will not use your locally modified version of Gitaly. To use a custom Gitaly version in CI you need to update GITALY_SERVER_VERSION as described at the beginning of this paragraph.

To use a different Gitaly repository, e.g., if your changes are present on a fork, you can specify a GITALY_REPO_URL environment variable when running tests: 

GITALY_REPO_URL=https://gitlab.com/nick.thomas/gitaly bundle exec rspec spec/lib/gitlab/git/repository_spec.rb

If your fork of Gitaly is private, you can generate a Deploy Token and specify it in the URL: 

GITALY_REPO_URL=https://gitlab+deploy-token-1000:token-here@gitlab.com/nick.thomas/gitaly bundle exec rspec spec/lib/gitlab/git/repository_spec.rb

To use a custom Gitaly repository in CI, for instance if you want your GitLab fork to always use your own Gitaly fork, set GITALY_REPO_URL as a CI environment variable.

Use a locally modified version of Gitaly RPC client

If you are making changes to the RPC client, such as adding a new endpoint or adding a new parameter to an existing endpoint, follow the guide for Gitaly proto. After pushing the branch with the changes (new-feature-branch, for example):

  1. Change the gitaly line in the Rails’ Gemfile to: 

    gem 'gitaly', git: 'https://gitlab.com/gitlab-org/gitaly.git', branch: 'new-feature-branch'

  2. Run bundle install to use the modified RPC client.

Return to Development documentation

Wrapping RPCs in Feature Flags

Here are the steps to gate a new feature in Gitaly behind a feature flag.

Gitaly

  1. Create a package scoped flag name: 

    var findAllTagsFeatureFlag = "go-find-all-tags"

  2. Create a switch in the code using the featureflag package: 

    if featureflag.IsEnabled(ctx, findAllTagsFeatureFlag) {
  // go implementation
} else {
  // ruby implementation
}

  3. Create Prometheus metrics: 

    var findAllTagsRequests = prometheus.NewCounterVec(
  prometheus.CounterOpts{
    Name: "gitaly_find_all_tags_requests_total",
    Help: "Counter of go vs ruby implementation of FindAllTags",
  },
  []string{"implementation"},
)
   
func init() {
  prometheus.Register(findAllTagsRequests)
}
   
if featureflag.IsEnabled(ctx, findAllTagsFeatureFlag) {
  findAllTagsRequests.WithLabelValues("go").Inc()
  // go implementation
} else {
  findAllTagsRequests.WithLabelValues("ruby").Inc()
  // ruby implementation
}

  4. Set headers in tests: 

    import (
  "google.golang.org/grpc/metadata"
   
  "gitlab.com/gitlab-org/gitaly/internal/featureflag"
)
   
//...
   
md := metadata.New(map[string]string{featureflag.HeaderKey(findAllTagsFeatureFlag): "true"})
ctx = metadata.NewOutgoingContext(context.Background(), md)
   
c, err = client.FindAllTags(ctx, rpcRequest)
require.NoError(t, err)

GitLab Rails

  1. Test in a Rails console by setting the feature flag:

    Note: Pay attention to the name of the flag and the one used in the Rails console. There is a difference between them (dashes replaced by underscores and name prefix is changed). Make sure to prefix all flags with gitaly_.
    Feature.enable('gitaly_go_find_all_tags')

Testing with GDK

To be sure that the flag is set correctly and it goes into Gitaly, you can check the integration by using GDK:

  1. The state of the flag must be observable. To check it, you need to enable it by fetching the Prometheus metrics:
    1. Navigate to GDK’s root directory.
    2. Make sure you have the proper branch checked out for Gitaly.
    3. Recompile it with make gitaly-setup and restart the service with gdk restart gitaly.
    4. Make sure your setup is running: gdk status | grep praefect.
    5. Check what configuration file is used: cat ./services/praefect/run | grep praefect value of the -config flag
    6. Uncomment prometheus_listen_addr in the configuration file and run gdk restart gitaly.
  2. Make sure that the flag is not enabled yet:
    1. Perform whatever action is required to trigger your changes (project creation, submitting commit, observing history, etc.).

    2. Check that the list of current metrics has the new counter for the feature flag: 

      curl --silent http://localhost:9236/metrics | grep go_find_all_tags
  3. Once you observe the metrics for the new feature flag and it increments, you can enable the new feature:
    1. Navigate to GDK’s root directory.

    2. Start a Rails console: 

      bundle install && bundle exec rails console

    3. Check the list of feature flags: 

      Feature::Gitaly.server_feature_flags

      It should be disabled "gitaly-feature-go-find-all-tags"=>"false".

    4. Enable it: 

      Feature.enable('gitaly_go_find_all_tags')
    5. Exit the Rails console and perform whatever action is required to trigger your changes (project creation, submitting commit, observing history, etc.).

    6. Verify the feature is on by observing the metrics for it: 

      curl --silent http://localhost:9236/metrics | grep go_find_all_tags