Fast lookup of authorized SSH keys in the database
Introduced in GitLab Starter 9.3.
Available in GitLab Community Edition 10.4.
Regular SSH operations become slow as the number of users grows because OpenSSH
searches for a key to authorize a user via a linear search. In the worst case,
such as when the user is not authorized to access GitLab, OpenSSH will scan the
entire file to search for a key. This can take significant time and disk I/O,
which will delay users attempting to push or pull to a repository. Making
matters worse, if users add or remove keys frequently, the operating system may
not be able to cache the authorized_keys file, which causes the disk to be
accessed repeatedly.
GitLab Shell solves this by providing a way to authorize SSH users via a fast, indexed lookup in the GitLab database. This page describes how to enable the fast lookup of authorized SSH keys.
Warning: OpenSSH version 6.9+ is required because
AuthorizedKeysCommandmust be able to accept a fingerprint. These instructions will break installations using older versions of OpenSSH, such as those included with CentOS 6 as of September 2017. If you want to use this feature for CentOS 6, follow the instructions on how to build and install a custom OpenSSH package before continuing.
Setting up fast lookup via GitLab Shell
GitLab Shell provides a way to authorize SSH users via a fast, indexed lookup to the GitLab database. GitLab Shell uses the fingerprint of the SSH key to check whether the user is authorized to access GitLab.
Add the following to your sshd_config file. This is usually located at
/etc/ssh/sshd_config, but it will be /assets/sshd_config if you're using
Omnibus Docker:
AuthorizedKeysCommand /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell-authorized-keys-check git %u %k
AuthorizedKeysCommandUser gitReload OpenSSH:
# Debian or Ubuntu installations
sudo service ssh reload
# CentOS installations
sudo service sshd reloadConfirm that SSH is working by removing your user's SSH key in the UI, adding a new one, and attempting to pull a repo.
Warning: Do not disable writes until SSH is confirmed to be working perfectly, because the file will quickly become out-of-date.
In the case of lookup failures (which are common), the authorized_keys
file will still be scanned. So git SSH performance will still be slow for many
users as long as a large file exists.
You can disable any more writes to the authorized_keys file by unchecking
Write to "authorized_keys" file in the Application Settings of your GitLab
installation.
Again, confirm that SSH is working by removing your user's SSH key in the UI, adding a new one, and attempting to pull a repo.
Then you can backup and delete your authorized_keys file for best performance.
How to go back to using the authorized_keys file
This is a brief overview. Please refer to the above instructions for more context.
- Rebuild the
authorized_keysfile - Enable writes to the
authorized_keysfile in Application Settings - Remove the
AuthorizedKeysCommandlines from/etc/ssh/sshd_configor from/assets/sshd_configif you are using Omnibus Docker. - Reload sshd:
sudo service sshd reload - Remove the
/opt/gitlab-shell/authorized_keysfile
Compiling a custom version of OpenSSH for CentOS 6
Building a custom version of OpenSSH is not necessary for Ubuntu 16.04 users, since Ubuntu 16.04 ships with OpenSSH 7.2.
It is also unnecessary for CentOS 7.4 users, as that version ships with
OpenSSH 7.4. If you are using CentOS 7.0 - 7.3, we strongly recommend that you
upgrade to CentOS 7.4 instead of following this procedure. This should be as
simple as running yum update.
CentOS 6 users must build their own OpenSSH package to enable SSH lookups via the database. The following instructions can be used to build OpenSSH 7.5:
-
First, download the package and install the required packages:
sudo su - cd /tmp curl --remote-name https://mirrors.evowise.com/pub/OpenBSD/OpenSSH/portable/openssh-7.5p1.tar.gz tar xzvf openssh-7.5p1.tar.gz yum install rpm-build gcc make wget openssl-devel krb5-devel pam-devel libX11-devel xmkmf libXt-devel -
Prepare the build by copying files to the right place:
mkdir -p /root/rpmbuild/{SOURCES,SPECS} cp ./openssh-7.5p1/contrib/redhat/openssh.spec /root/rpmbuild/SPECS/ cp openssh-7.5p1.tar.gz /root/rpmbuild/SOURCES/ cd /root/rpmbuild/SPECS -
Next, set the spec settings properly:
sed -i -e "s/%define no_gnome_askpass 0/%define no_gnome_askpass 1/g" openssh.spec sed -i -e "s/%define no_x11_askpass 0/%define no_x11_askpass 1/g" openssh.spec sed -i -e "s/BuildPreReq/BuildRequires/g" openssh.spec -
Build the RPMs:
rpmbuild -bb openssh.spec -
Ensure the RPMs were built:
ls -al /root/rpmbuild/RPMS/x86_64/You should see something as the following:
total 1324 drwxr-xr-x. 2 root root 4096 Jun 20 19:37 . drwxr-xr-x. 3 root root 19 Jun 20 19:37 .. -rw-r--r--. 1 root root 470828 Jun 20 19:37 openssh-7.5p1-1.x86_64.rpm -rw-r--r--. 1 root root 490716 Jun 20 19:37 openssh-clients-7.5p1-1.x86_64.rpm -rw-r--r--. 1 root root 17020 Jun 20 19:37 openssh-debuginfo-7.5p1-1.x86_64.rpm -rw-r--r--. 1 root root 367516 Jun 20 19:37 openssh-server-7.5p1-1.x86_64.rpm -
Install the packages. OpenSSH packages will replace
/etc/pam.d/sshdwith its own version, which may prevent users from logging in, so be sure that the file is backed up and restored after installation:timestamp=$(date +%s) cp /etc/pam.d/sshd pam-ssh-conf-$timestamp rpm -Uvh /root/rpmbuild/RPMS/x86_64/*.rpm yes | cp pam-ssh-conf-$timestamp /etc/pam.d/sshd -
Verify the installed version. In another window, attempt to login to the server:
ssh -v <your-centos-machine>You should see a line that reads: "debug1: Remote protocol version 2.0, remote software version OpenSSH_7.5"
If not, you may need to restart sshd (e.g.
systemctl restart sshd.service). -
IMPORTANT! Open a new SSH session to your server before exiting to make sure everything is working! If you need to downgrade, simple install the older package:
# Only run this if you run into a problem logging in yum downgrade openssh-server openssh openssh-clients