Introduce proxy auth support by aylei · Pull Request #8751 · skypilot-org/skypilot

aylei · 2026-01-29T23:42:46Z

Previously, the API server only supported nginx + oauth2-proxy for SSO authentication on the ingress. Users deploying behind other authentication proxies had limited options.

This PR extends the authentication system to trust any external proxy that sets a user identity header, supporting two kinds of header format:

plaintext: the trusted header will be used as the identity
jwt: the trusted header is a JWT token, in this case an optional jwt_identity_claim can be specified (default to sub) as the user identity as the request

When oauth.externalProxy is enabled, all the requests are assumed to have been authenticated by the proxy in front of the skypilot API server, it is the user's duty to ensure the authentication scheme on the proxy is secure and no requests can be sent to API server bypassing the proxy.

Usage: refer to the docs/source/reference/api-server/helm-values-spec.rst

Tested (run the relevant ones):

Any manual or new tests for this PR (please specify below)

Local test:

$ cat ~/.sky/.server.yaml
auth:
  external_proxy:
    enabled: true
    header_name: X-WEBAUTH-TOKEN
    header_format: jwt
    jwt_identity_claim: email
$ JWT_TOKEN=$(python3 -c "import jwt; print(jwt.encode({'email': 'fromjwt@example.com'}, 'secret', algorithm='HS256'))")
$ curl -H "X-WEBAUTH-TOKEN: $JWT_TOKEN" http://localhost:46580/users
[{"id":"skypilot-system","name":"skypilot-system","created_at":1769627590,"role":"admin"},{"id":"57339c81","name":"aylei","created_at":1769627592,"role":"admin"},{"id":"e153a762","name":"test11111@example.com","created_at":1769732651,"role":"admin"},{"id":"d350dfa5","name":"test2222@example.com","created_at":1769732812,"role":"admin"},{"id":"ee9cd058","name":"fromjwt@example.com","created_at":1769753337,"role":"admin"}]

Test with auth.oauth.enabled==true on existing deployment, does not break existing OAuth2
Test with ingress auth new deployment, works as expected

Code formatting: install pre-commit (auto-check on commit) or bash format.sh
New unit test cases
All smoke tests: /smoke-test (CI) or pytest tests/test_smoke.py (local)
Relevant individual tests: /smoke-test -k test_name (CI) or pytest tests/test_smoke.py::test_name (local)
Backward compatibility: /quicktest-core (CI) or pytest tests/smoke_tests/test_backward_compat.py (local)

Signed-off-by: Aylei <rayingecho@gmail.com>

gemini-code-assist · 2026-01-29T23:43:10Z

Summary of Changes

Hello @aylei, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly extends the authentication capabilities of the SkyPilot API server by integrating support for external authentication proxies. This feature allows deployments to leverage existing identity providers and proxy solutions (like AWS ALB with Cognito or Azure Front Door) by trusting user identity information passed through specific HTTP headers. The changes encompass new Helm chart configurations for easy deployment, robust server-side logic to parse and validate user identities from these headers (including JWT tokens), and crucial validation rules to ensure compatibility and prevent conflicts with other authentication methods.

Highlights

External Proxy Authentication Configuration: Introduced a new auth.externalProxy section in Helm charts (values.yaml) to allow flexible configuration of external authentication proxies, including options for enabling, header name, header format (plaintext or JWT), and JWT identity claim.
Server-Side Identity Extraction: Implemented server-side logic to process user identities from HTTP headers, supporting both plaintext values and decoding of JWT tokens without verification for trusted proxy scenarios.
Mutual Exclusivity Validation: Added validation checks (skypilot.validateExternalProxyConfig) to ensure that external proxy authentication is mutually exclusive with existing OAuth and ingress OAuth2 proxy authentication methods, preventing conflicting configurations.
Dedicated Server Configuration File: Refactored server configuration to utilize a dedicated server.yaml file (generated via server-config.yaml Helm template), improving manageability and enabling advanced authentication settings, with backward compatibility for legacy environment variables.
Enhanced Unit Test Coverage: Expanded unit test coverage for the new external proxy configuration loading, identity extraction mechanisms, and various scenarios including defaults, overrides, and backward compatibility.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces support for external proxy authentication, which is a great addition for flexible deployment scenarios. The changes are well-implemented across the Helm charts and the server-side Python code. I appreciate the attention to backward compatibility and the comprehensive unit tests. I have a couple of suggestions to improve performance by caching the configuration loading.

Signed-off-by: Aylei <rayingecho@gmail.com>

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Signed-off-by: Aylei <rayingecho@gmail.com>

aylei · 2026-01-30T07:12:12Z

/smoke-test

DanielZhangQD

Thanks! @aylei

DanielZhangQD · 2026-01-30T07:51:47Z

+        # on the ingress). So:
+        # - If any of the built-in auth schemes is enabled, disable it since
+        # built-in auth is exclusive with auth on the ingress
+        # - Otherwise we enable it since user should have at least one ingress


How does this work with the basic auth on ingress and oauth on ingress cases?

Oh, the external proxy should take the role of the nginx ingress in previous deployments.

DanielZhangQD

LGTM!

DanielZhangQD · 2026-01-30T08:24:12Z

The failed smoke test cases are not introduced by this PR.

Introduce proxy auth support

ecbd42f

Signed-off-by: Aylei <rayingecho@gmail.com>

gemini-code-assist Bot reviewed Jan 29, 2026

View reviewed changes

Comment thread sky/server/config.py

Comment thread sky/server/config.py

aylei and others added 6 commits January 29, 2026 15:54

Helm docs

4c04ac5

Signed-off-by: Aylei <rayingecho@gmail.com>

Update sky/server/config.py

86ea679

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Update sky/server/config.py

52641f2

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

Simplify the doc

81aeb06

Signed-off-by: Aylei <rayingecho@gmail.com>

Format

aff45fd

Signed-off-by: Aylei <rayingecho@gmail.com>

Merge branch 'master' into external-proxy-auth-support

4e83de3

Michaelvll added this to the 0.11.2 milestone Jan 30, 2026

Refine UT

279f43f

Signed-off-by: Aylei <rayingecho@gmail.com>

aylei marked this pull request as ready for review January 30, 2026 07:18

aylei requested review from DanielZhangQD and cg505 January 30, 2026 07:19

DanielZhangQD reviewed Jan 30, 2026

View reviewed changes

DanielZhangQD approved these changes Jan 30, 2026

View reviewed changes

DanielZhangQD merged commit 270e643 into master Jan 30, 2026
23 of 24 checks passed

DanielZhangQD deleted the external-proxy-auth-support branch January 30, 2026 08:25

This was referenced Jan 30, 2026

OAuth2 proxy authentication fails with Traefik ForwardAuth due to internal re-validation #8711

Closed

[AuthN] Support formatted auth headers such as x-amzn-oidc-data #6136

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Introduce proxy auth support#8751

Introduce proxy auth support#8751
DanielZhangQD merged 8 commits into
masterfrom
external-proxy-auth-support

aylei commented Jan 29, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Jan 29, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

aylei commented Jan 30, 2026

Uh oh!

DanielZhangQD left a comment

Uh oh!

DanielZhangQD Jan 30, 2026

Uh oh!

DanielZhangQD Jan 30, 2026

Uh oh!

DanielZhangQD left a comment

Uh oh!

DanielZhangQD commented Jan 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

aylei commented Jan 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot commented Jan 29, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

aylei commented Jan 30, 2026

Uh oh!

DanielZhangQD left a comment

Choose a reason for hiding this comment

Uh oh!

DanielZhangQD Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

DanielZhangQD Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

DanielZhangQD left a comment

Choose a reason for hiding this comment

Uh oh!

DanielZhangQD commented Jan 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aylei commented Jan 29, 2026 •

edited

Loading