Sourcegraph 3.4: Performance and configuration enhancements for managing 30,000+ repositories

Ryan Blunden

Crhstina Forney

May 20, 2019

Sourcegraph is the standard developer platform for code search and navigation at many of the largest and most exacting technology companies. With Sourcegraph, every company can get access to the same kind of tools that Google and Facebook developers use every day.

We're excited to announce Sourcegraph 3.4. This release makes Sourcegraph more flexible and configurable to meet the needs of our large enterprise customers, such as Uber, Lyft and Yelp. Our customers rely on Sourcegraph to search across 30,000+ repositories, with 1,000s of daily users on single instances.

🗺 Browser extension now supports repository path mapping Get shorter repository URLs in Sourcegraph.

🛠 Code host sync improvements for AWS CodeCommit and Gitolite Easily manage thousands of repositories, plus AWS CodeCommit gets a new gitCredentials field.

🗂 Optional loading of configuration from the file system or Kubernetes ConfigMap For site admins, instead of saving configuration to the database.

⏰ Configurable upgrade notifications for patch releases

🔎 Higher information density in search results

📝 Changelog

🎖️ Thank you

Deploy or upgrade: Local | AWS | DigitalOcean | Kubernetes cluster

Browser extension now supports repository path mapping

View on Vimeo

If your code host's hostname is long, such as githubenterprise.mycompany.internal, then your Sourcegraph URLs will be long (e.g. https://sourcegraph.mycompany.internal /githubenterprise.mycompany.internal /myteam/myproject). To shorten these URLs, you can use the repositoryPathPattern external service configuration property.

Sourcegraph 3.4 fixes a problem where the browser extension didn't work if you used repositoryPathPattern. If you had been holding off on using the browser extension due to this problem, it will now work.

Code host sync improvements for AWS CodeCommit and Gitolite

Sourcegraph 3.3 saw the introduction of config-based repository selection (for GitHub, GitLab and Bitbucket Server). Now, Sourcegraph 3.4 brings the same model to AWS CodeCommit, and Gitolite.

As a result, AWS CodeCommit and Gitolite get a new exclude field, plus AWS CodeCommit now supports the gitCredentials option for using a static username and password tied to an IAM user.

See the changelog for more details

Optional loading of configuration from the file system or Kubernetes ConfigMap

Some teams for whom Sourcegraph is a critical piece of infrastructure want to check their Sourcegraph configuration into version control.

For these teams, site configuration, critical configuration, and external services configuration can be loaded from the file system or Kubernetes ConfigMap if the following environment variables are set on the server:

CRITICAL_CONFIG_FILE=critical.jsonSITE_CONFIG_FILE=site.jsonEXTSVC_CONFIG_FILE=extsvc.json

For cluster (Kubernetes) deployments, an admin is responsible for creating the ConfigMap resources and setting these environment variables to refer to the ConfigMaps' files on disk.

Configurable upgrade notifications for patch releases

View on Vimeo

Sourcegraph detects when a new version is available and shows a dismissible notification to site admins. In Sourcegraph 3.4, site admins can now control if they want to see patch release notifications by adding the following to settings:

{ ... “alerts.showPatchUpdates": false, ...}

Upgrade notifications are always shown for major (e.g. 4.0) or minor (e.g. 3.x) releases.

Higher information density in search results

We're tightening up the Sourcegraph UI to remove unnecessary whitespace so you can see more information without scrolling. This release ships with a cleaner search results interface.

File search results (before and after):

Text search results (before and after):

3.4 changelog

Added

When repositoryPathPattern is configured, paths from the full long name will redirect to the configured name. Extensions will function with the configured name. repositoryPathPattern allows administrators to configure "nice names". For example sourcegraph.example.com/github.com/foo/bar can configured to be sourcegraph.example.com/gh/foo/bar with "repositoryPathPattern": "gh/{nameWithOwner}". (#462)
Admins can now turn off site alerts for patch version release updates using the alerts.showPatchUpdates setting. Alerts will still be shown for major and minor version updates.
The new gitolite.exclude setting in Gitolite external service config allows you to exclude specific repositories by their Gitolite name so that they won't be mirrored. Upon upgrading, previously "disabled" repositories will be automatically migrated to this exclusion list.
The new aws_codecommit.exclude setting in AWS CodeCommit external service config allows you to exclude specific repositories by their AWS name or ID so that they won't be synced. Upon upgrading, previously "disabled" repositories will be automatically migrated to this exclusion list.
Added a new, required aws_codecommit.gitCredentials setting to the AWS CodeCommit external service config. These Git credentials are required to create long-lived authenticated clone URLs for AWS CodeCommit repositories. For more information about Git credentials, see the AWS CodeCommit documentation: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_ssh-keys.html#git-credentials-code-commit. For detailed instructions on how to create the credentials in IAM, see this page: https://docs.aws.amazon.com/codecommit/latest/userguide/setting-up-gc.html
Added support for specifying a URL formatted gitolite.host setting in Gitolite external service config (e.g. ssh://[email protected]:2222/), in addition to the already supported SCP like format (e.g [email protected])
Added support for overriding critical, site, and external service configurations via files. Specify CRITICAL_CONFIG_FILE=critical.json, SITE_CONFIG_FILE=site.json, and/or EXTSVC_CONFIG_FILE=extsvc.json on the frontend container to do this.

Changed

Kinds of external services in use are now included in server pings.
Bitbucket Server: An actual Bitbucket icon is now used for the jump-to-bitbucket action on repository pages instead of the previously generic icon.
Default config for GitHub, GitHub Enterprise, GitLab, Bitbucket Server, and AWS Code Commit external services has been revised to make it easier for first time admins.

Removed

Fields related to Repository enablement have been deprecated. Mutations are now NOOPs, and for repositories returned the value is always true for Enabled. The enabled field and mutations will be removed in 3.6. Mutations: setRepositoryEnabled, setAllRepositoriesEnabled, updateAllMirrorRepositories, deleteRepository. Query parameters: repositories.enabled, repositories.disabled. Field: Repository.enabled.
Global saved searches are now deprecated. Any existing global saved searches have been assigned to the Sourcegraph instance's first site admin's user account.
The search.savedQueries configuration option is now deprecated. Existing entries remain in user and org settings for backward compatibility, but are unused as saved searches are now stored in the database.

Fixed

Fixed a bug where submitting a saved query without selecting the location would fail for non-site admins (#3628).
Fixed settings editors only having a few pixels height.
Fixed a bug where browser extension and code review integration usage stats were not being captured on the site-admin Usage Stats page.
Fixed an issue where in some rare cases PostgreSQL starting up slowly could incorrectly trigger a panic in the frontend service.
Fixed an issue where the management console password would incorrectly reset to a new secure one after a user account was created.
Fixed a bug where gitserver would leak file descriptors when performing common operations.
Substantially improved the performance of updating Bitbucket Server external service configurations on instances with thousands of repositories, going from e.g. several minutes to about a minute for ~20k repositories (#4037).
Fully resolved the search performance regression in v3.2.0, restoring performance of search back to the same levels it was before changes made in v3.2.0.
Fix a bug where using a repo search filter with the prefix github.com only searched for repos whose name starts with github.com, even though no ^ was specified in the search filter. (#4103)

The changelog for this and previous releases is available on GitHub.

Thank you

Thank you to the many people who contributed to Sourcegraph since the last release!

‍

Ready to accelerate
how you build software?

Use Sourcegraph to industrialize your software development