fix: retry graphql queries that failed with 502

• Addresses errors similar to: failed to prepare workspace: failed to create ext4 filesystem in backing file: "mke2fs 1.45.5 (07-Jan-2020)\nThe file losetup: /var/lib/firecracker/loop-devices/workspace-loop-47749-3763347387: Warning: file is smaller than 512 bytes; the loop device may be useless or invisible for system tools.\n/dev/loop13 does not exist and no size was specified.\n": exit status 1
• Affects reading and using the environment variable EXECUTOR_FIRECRACKER_DISK_SPACE
  Backport 22db2a97a0393cdd233a92da7ade96cd59bb99ef from #2562

Special uppercase regex tokens are no longer inverted for repository name search. eg repo:foo\B correctly interprets \B as not at ASCII word boundary. Previously it would transform the expression into repo:foo\b.

• Adds configurable limits for file size and parsing time for syntactic indexing

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

• release: nightly release pipeline is working

Sourcegraph now supports App-level GitHub App connections that aren't tied to a specific Installation ID.

feat(batches): more telemetry events

feat: extend prompt tags with pagination and increased limit

feat: editing tags on prompts

feat: management page prompt categories

feat: add tags as filter to prompts and their navigation

feat: add prompt labels to the GraphQL API

feat: calculate contributor stats

• Also validates KUBERNETES_RUN_AS_USER and KUBERNETES_RUN_AS_GROUP.
• Must be integers in the range [0, 2147483647].

Properly report symbols for TypeScript type aliases

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

Narrowed down the stopwords list used in Cody context, to fix an issue where we could miss highly relevant files.

Fixes a bug in Cody context where airgapped instances would attempt to use the reranker model, which increased the context retrieval latency.

N/A

N/A

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

N/A

N/A

• refactor and fix bugs in release test

<details>

➜  sourcegraph git:(wg/rel/post-release-5-10-fixup) ✗ bazel run //testing/tools/upgradetest:sh_upgradetest_run -- all -svs 5.9.0 -mvs 5.8.0 -avs 5.8.0 -pv 5.10.2832
INFO: Analyzed target //testing/tools/upgradetest:sh_upgradetest_run (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //testing/tools/upgradetest:sh_upgradetest_run up-to-date:
  bazel-bin/testing/tools/upgradetest/sh_upgradetest_run
Aspect @@rules_rust//rust/private:clippy.bzl%rust_clippy_aspect of //testing/tools/upgradetest:sh_upgradetest_run up-to-date (nothing to build)
INFO: Elapsed time: 1.165s, Critical Path: 0.72s
INFO: 5 processes: 1 internal, 4 darwin-sandbox.
INFO: Build completed successfully, 5 total actions
INFO: Running command line: bazel-bin/testing/tools/upgradetest/sh_upgradetest_run testing/tools/upgradetest/go_upgradetest_/go_upgradetest cmd/migrator/image_tarball.sh cmd/frontend/no_client_bundle.image_tarball.sh docker-images/postgresql-16/image_tarball.sh docker-images/postgresql-16-codeinsights/image_tarball.sh internal/database/_codeinsights_squashed.sql internal/database/_codeinsights_tenant_data_squashed.sql internal/database/_codeintel_squashed.sql internal/database/_codeintel_tenant_data_squashed.sql internal/database/_frontend_squashed.sql internal/database/_frontend_tenant_data_squashed.sql internal/database/_schema.codeinsights.json internal/database/_schema.codeinsights.md internal/database/_schema.codeintel.json internal/database/_schema.codeintel.md internal/database/_schema.json internal/database/_schema.md all -svs 5.9.0 -mvs 5.8.0 -avs 5.8.0 -pv 5.10.2832
Loaded image: migrator:candidate
Loaded image: frontend:candidate
Loaded image: postgresql-16:candidate
Loaded image: postgresql-16-codeinsights:candidate
👉 Upgrade test ...
Latest stable release version:  5.10.2832
Latest minor version:  5.10.0
Target version:  5.10.2832
Migrator image used to upgrade:  sourcegraph/migrator:5.10.2832
Standard Versions: [5.9.0]
Multiversion Versions: [5.8.0]
Autoupgrade Versions: [5.8.0]
auto:  5.8.0
mvu:  5.8.0
std:  5.9.0
--- 🕵️  Standard Upgrade Tests:
✅ 5.9.0 Passed -- 1m16.366600375s
--- 🕵️  Multiversion Upgrade Tests:
✅ 5.8.0 Passed -- 1m17.939247958s
--- 🕵️  Auto Upgrade Tests:
✅ 5.8.0 Passed -- 1m24.54765675s

</details>
Backport 8dc831953f43bbe9f6b9cb5467ef263b757425ac from #2374

fix: remove surplus typename

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c
Backport 3f079a57978179aa2ad3f310195346a8c574f9ce from #2349

We no longer log the postgresql DSN when it changes. Previously this could contain the secret PGPASSWORD.
Backport 2bd9c09969fff536c6fb29108656658d3f875047 from #2272

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c
Backport 964eceae584e4eff56df9711da68fef7047cbc40 from #2254

fix(rel): change pg_upgrade behavior to use copy instead of hard links
Backport 13caa807d96dd4b5dd872fec2a38322d1888134a from #2192

fix(rel): address issue with readiness check outputting passwords to logs.
Backport 655a739e8e791d7aab1977d2c26543a3a4871809 from #2132

Disabled an indexed search optimization which would skip files accidentally (ZOEKT_DISABLE_GOGIT_OPTIMIZATION=true).
Backport 34ada948bdcee3d75499c98f4db5c32986943e88 from #2050

Cody context now incorporates filename information in reranking, improving context quality when the reranker is enabled.

• adds systemPreInstruction to the modelConfiguration key in site admin config that allows injecting a prelude prompt into every chat request for an enterprise

• Cody Gateway: add support for OpeanAI predicted outputs

The GitHub app authentication package now supports authenticating a GitHub app via the OAuth client ID following the announcement of https://github.blog/changelog/2024-05-01-github-apps-can-now-use-the-client-id-to-fetch-installation-tokens/

A new GraphQL query, GithubAppRepositoriesForInstallation, has been added that provides a paginated list of all the GitHub repositories that are accessible to the GitHub app with the provided installation id.

A new graphql endpoint has been added, GithubAppInstallationsForUser, that returns installation information for the global GitHub multi tenant app when running in multitenant mode.

• Cody Gateway: add gpt-4o-mini to Cody Gateway allow list
• Cody Gateway: add gpt-4o-mini to DotCom models list

Feat

Dev

• Adding sg command to request Entitle bundles #1370

The Github API client's GetUserInstallations route, which lists of GitHub App installations the user has access to, now has pagination support.

A simple helper routine to the multitenantenv package that automatically populates a github app struct with the provided validated credentials.

The routing logic for multitenant mode now has a new route that uses the "last seen tenant" cookie to route github app login authorization callbacks to the appropriate tenant.

feat(rel): Add Postgresql 16 and Postgresql 16 codeinsights images to published image list.

feat(rel): Add self updating to Postgres 16 codeinsights db image.

feat(rel): Add self updating to Postgres 16 container image.

feat(rel): Add Postgres 16 CodeInsights Wolfi image

feat(rel): Add Postgres 16 Wolfi image

• refactor upgradetest
• introduce proper handling of the postgres version upgrade
  Backport 9ccdf4200e3e08cea56bffe5779ca8a6cda2909c from #1894

• support get, list, and delete workspaces to sg

A new worker job has been added that updates the database with the credentials for the global github app when running in multitenant mode.

N/A

• Added dynamic filters and the ability to aggregate by repo metadata and repo topic

• Search filters sidebar is now collapsible

• Render .mdx files as markdown

We sometimes would emit patternType:regex instead of patternType:regexp. We now always do regexp as well as treating regex as an alias for regexp.
Backport a095b39ac39cfcbe3526ecf85ed6d50cb5fa3d9d from #1808

fix(batches): titles now break over multiple lines if they are too long

Code insights should show correct line counts when enhanced language detection is turned off (note: this setting is on by default).

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

The github_apps table has removed an erroneous unique constraint that accidentally incorporated an app's slug (which is mutable) as part of the unique identifier for an app.

The logic in the new multitenant GraphQL resolvers has been reworked to use the new helper methods introduced in the Multitenant GitHub app configuration logic introduced in https://app.graphite.dev/github/pr/sourcegraph/sourcegraph/1758/.

This change reduces the risk of secrets being emitted in executor logs even if there are bugs in the secret redaction logic.

NA
Backport a943412a99852332f921b52a0ee3179dc3331d20 from #1958

fix(rel): Add vacuum to Postgres upgrade process

N/A

• add migrations to handle database drift caused during postgres 12 to postgres 16 upgrade

• Fix bug in local upgrade test preventing final stage drift check via private monorepo

• When cloning Perforce depots, Sourcegraph will now decode encoded paths correctly (paths that include characters like @ and #)

When using an instance with a Cody-only license, accessing the home page now correctly redirects you to /cody/dashboard, instead of a non-existent /cody page.
Backport a4cb5a0723bad18e1c215d81231db457d1abfbdb from #1621

feat(batches): show name of additional fields that cause errors

Long-term local retention of user telemetry as 'event logs' can now be disabled entirely via the telemetry: { disableLocalEventLogs } site configuration.

• Adds cost estimates to the ModelConfiguration returned by .api/llm/supported-models.json
• Adds llmTokenUsageCostEstimate telemetry value with estimates of each requests LLM cost in pennies

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

• Add the sourcegraph API client user-agent to telemetry payload types under source.server.server_client.user_agent.

• Propagates new X-Sourcegraph-API-Client-Name and X-Sourcegraph-API-Client-Version headers to request clients

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

• New OpenAPI API docs within the instance at the URL $SRC_ENDPOINT/api/openapi/public. Alternatively, reach the page at "User > Settings > OpenAPI Reference".

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

• The /.api/completions/stream API now includes used input/output tokens and the upstream model name in the response body when using the query parameter api-version=5 or higher. Does not apply to: Vertex Anthropic, Azure OpenAI (mo model name, and no token usage in streaming response), Gemini (no model name in streaming response)
• LLM chat completions now support stream: false when using Azure OpenAI and OpenAI-compatible providers.

• The raw HTTP API now accepts the query paramter format=cody or header Accept: application/cody to render files and directories as context items for Cody. Example request path: /github.com/sourcegraph/cody/-/raw/agent/src/index.ts?format=cody.

• add library that writes to TEST_INFRASTRUCTURE_FAILURE_FILE if it is defined for test failures

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

Cody Gateway: add new claude-3-5-haiku-latest & claude-3-5-opus-latest models to Cody Gateway allow list Cody Gateway: add new claude-3-5-haiku-latest dotcom models list
Backport 11e7481ba3c810ae5b47d32b8cf32066e2f0b2bb from #1470

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

The sub repository permissions implementation now emits logs that describe the rules that were evaluated for a given request whenever tracing is enabled. (We limit these logs for only when tracing is enabled since they can be quite verbose).

• Added embeddable page for file snippets

Prevents default browser shortcuts from modifying the file content locally.

This fixes a bug where we would apply a phrase boost if a content: filter was specified with keyword search enabled. This led to inconsistent behavior (regexp vs keyword search) and it also rendered the content: filter ineffective, because we ran a general text search instead of a just a content search.

Fixes a bug where hovers were not triggered for less mainstream languages like F#, Standard ML, Visual Basic, Pkl, Hack, MATLAB etc.

Introduces new limits for auto-indexing inference to reduce the risk of continuously growing auto-indexing queues.

• The number of jobs spawned per round of auto-indexing inference per repo is capped (default: 100) to reduce risk of clogging of auto-indexing queues. Excess jobs will be discarded.
• The number of paths inspected for a single round of auto-indexing inference per repo is capped (default: 500) to reduce risk of timeouts. Excess paths will be discarded.

Fixes handling of SCIP uploads when the hash passed to the -commit flag of src-cli corresponds to the hash of an annotated tag instead of the hash of a commit. Previously, these uploads were not accessible for code navigation.

Fixes a bug where the retention tab for precise indexes would show a nil pointer exception instead of retention information.

fix(batches): enable request splitting by default

fix(batches): handle spec being nil for suffix salt

fix(batches): enable GQL request splitting via BATCH_CHANGES_REQUEST_SPLITTING

fix(batches): display "Deleted Namespace" instead of throwing an error

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

fix(code-monitor): fail monitors with missing/unverified primary email address

OPTIONAL; info at https://www.notion.so/sourcegraph/Writing-a-changelog-entry-dd997f411d524caabf0d8d38a24a878c

Added token usage information to .api/llm/chat/completions

Fixes a bug where logs and traces were sometimes missing key-value pairs recorded alongside errors.

NA

The documentation for gitserver's ReadDir method has been clarified to reflect that the return iterator's Name() method returns the full path of the file, as opposed to just the basename.

The documentation for the "exclude" section in the github code host connection schema has been clarified to explain that each block is OR'd together, and the expressions within each block are AND'd together.

• When there are multiple auth providers configured that point to the same GitHub URL, as can be the case when using private GitHub Apps, user external account tokens will now refresh correctly.

• GitHub code host connections using repositoryQuery will no longer delete repositories if the sync fails for reasons like a GitHub outage or a token expiring.

• Reduce peak memory required for Zoekt indexing
• Improve search ranking by using repo freshness as a scoring signal

With this change we rank repository search results by similarity. Before, repository search results were ordered by star count instead. This affects suggestions and repository search results. To disable this feature and return to the previous behavior, set { "experimentalFeatures": { "disableOrderBySimilarity": true}} in global settings.

The GraphQL endpoint search.results.repositories now returns the list of repositories in the order of the matches we found. Before, repositories were sorted by id.

• TypeScript: Fixes references to object properties in various places
• Ruby: Fixes references to Opus:: in Ruby tests

feat(batches): verbose error logging for batch syncer with BATCH_CHANGES_SYNCER_VERBOSE_ERROR_LOGGING

feat(batches): add experimental feature to add a fork name suffix to avoid name collisions

• /.api/completions/stream endpoint now accepts "file" and "repo" parts alongside "text" and "image_url"

• /.api/completions/stream now supports vision using the OpenAI-compatible base64 encoding of images. Example "content": [{"type": "image_url", "image_url": { "url": "data:image/png;base64,{{IMAGE_BASE64}}" } }]. Requires the query parameter api-version=3 or higher.

• enable aspect workflows ci telemetry

• generate constraints.go using bazel for onconflict linter

• add cody-gateway environments for subcommand live

• implement using the release-registry for the banner

• Remove upgradetest from the release promotion ci

• Allow PRs to block a release from going out 😈

• BitBucket Cloud, BitBucket Server, GitLab, Gerrit, Gitolite, AWS CodeCommit Azure DevOps code host connections now support configuring SSH key authentication from the UI instead of mounting from disk.

• GitHub and generic Git code host connections now support configuring SSH key authentication from the UI instead of mounting from disk.

• add pgvector to server and postgresql-12 images

• build and add pgvector extension

• Introduce a new experimental embeddings index and context retriever.

We have updated the client-side query validation to allow combining rev: filters with query-based search contexts.

For very large repositories, search jobs could time out while fetching the repository. Now, search jobs can fetch and search over repositories, even if they take a long time to fetch.

We added a new database index to speed up a janitor job which is run as part of Search Jobs. Before, the janitor job might have significantly delayed migrations during an upgrade.

The new usagesForSymbol API correctly searches repos whose names contain spaces or other special characters.

Go cross-repo navigation for tagged versions should work better when using scip-go for auto-indexing. Navigation based on SCIP indexes generated using scip-go in CI or cron jobs should be unaffected.

• Fixed incorrect executors version reporting which indicated 0.0.0+dev even though it was the correct executor AMI version running under the hood.

• override internal/version.version with x_defs at build time in executor

fix(batches): disabled "select all" checkbox if all nodes are already disabled

• Fix an issue where duplicate telemetry can be exported when the worker service is scaled horizontally

modelsAPIEnabled is always false for dotcom

• The /.api/llm/chat/completions endpoint will now return an empty string "" for finish_reason instead of the string "unknown_please_report_bug()".

• dump more information out about the branch if it is out of sync

• remove embeddings in promotion script

• fix PGDATASOURCE format when using unix sockets

• Fixes a bug that can cause errors when rendering many mermaid diagrams on a page.

This fixes a bug where we added "readme" too often to the context.
Backport 28ff196a663f537c6cb6340f976a91431509a90e from #582

fix(batches): workaround for a bug in GitHub
Backport aad3a04f8c93561a61c404e69132e70a22d0acba from #576

n/a

• Add a container registry deny list to complement the allow list.
• Configure in site config via "batchChanges.containerRegistryDenylist".
• Mutually exclusive with "batchChanges.containerRegistryAllowlist".

• New POST /.api/cody/context REST endpoint to retrieve a list of relevant source locations given a natural language query.

When searching an unindexed commit we would consult indexed commits for speeding up results. If our index contained a commit that no longer existed in git we would error out due to a regression in v5.4.5099. This is now fixed.

fix(batches): avoid "Name already exists on this account" from creating fork by fetching the repo when the error happens

Fixes a regression in Cody context where questions containing the word 'or' could return noisy or no results.

• LLM API endpoints (/.api/llm) now return JSON-encoded HTTP bodies for non-200 status codes.

• LLM API /.api/chat/completions now returns OpenAI-compatible finish_reason.

• For compatibility with OpenAI clients, it's possible to use Bearer TOKEN header with all API endpoints that start with the prefix /.api/llm.

• prevent web-sveltekit commands from running if there are untracked files under src/routes

repo-updater now emits logs that log the result of every code host sync.

• the cancel funcs used by sg commands are now wrapped in sync.OnceFunc to prevent duplicate execution

• sg: fix panic when using wolfi update-hashes
• sg: add deprecation notice for wolfi update-hashes

• sg - fix panic in Cloud Ephemeral listing when listing instances that are not Ephemeral

• ensure deployment / instance names are clamped in all places for cloud ephemeral