I am hoping to use OAuth2 with existing Traefik in our Kubernetes cluster.
Environment
- SkyPilot Helm chart version: 0.11.1
- Ingress controller: Traefik (not nginx-ingress)
- OAuth2 provider: Google OIDC
Problem
I am hoping to use Traefik instead of nginx-ingress. However,
ingress:
enabled: true
oauth2-proxy:
enabled: true
oidc-issuer-url: https://accounts.google.com
client-details-from-secret: hm-skypilot-secret
email-domain: example.com
use-https: true
session-store-type: rediswill create nginx-ingress, so I have to use:
ingress:
enabled: false
auth:
oauth:
enabled: trueThe issue is when using auth.oauth.enabled: true with Traefik instead of nginx-ingress, users are still prompted for username/password after successfully completing Google OAuth sign-in.
Configuration
Helm chart values.yaml:
ingress:
enabled: false
auth:
oauth:
enabled: true
oidc-issuer-url: https://accounts.google.com
client-details-from-secret: hm-skypilot-secret
email-domain: example.com
use-https: true
session-store-type: redis
ingress-nginx:
enabled: falseTraefik ForwardAuth Middleware:
# https://doc.traefik.io/traefik/reference/routing-configuration/http/middlewares/forwardauth/
---
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
name: hm-skypilot-forward-authentication-middleware
namespace: production-hm-skypilot
labels:
app.kubernetes.io/name: hm-skypilot-forward-authentication-middleware
app.kubernetes.io/part-of: production-hm-skypilot
spec:
forwardAuth:
address: http://skypilot-oauth2-proxy.production-hm-skypilot.svc:4180/oauth2/auth
trustForwardHeader: true
authResponseHeaders:
- X-Auth-Request-User
- X-Auth-Request-Email
- X-Auth-Request-Access-Token
- AuthorizationTraefik IngressRoute:
# https://doc.traefik.io/traefik/routing/providers/kubernetes-crd
---
apiVersion: traefik.io/v1alpha1
kind: IngressRoute
metadata:
name: hm-skypilot-ingress-route
namespace: production-hm-skypilot
annotations:
external-dns.alpha.kubernetes.io/hostname: skypilot.example.com
labels:
app.kubernetes.io/name: hm-skypilot-ingress-route
app.kubernetes.io/part-of: production-hm-skypilot
spec:
entryPoints:
- websecure
routes:
# OAuth2 proxy routes (/oauth2/sign_in, /oauth2/callback, etc.)
- match: Host(`skypilot.example.com`) && PathPrefix(`/oauth2`)
kind: Rule
services:
- name: skypilot-oauth2-proxy
port: 4180
# Service account token bypass (Bearer sky_* tokens validated by SkyPilot API)
- match: Host(`skypilot.example.com`) && HeadersRegexp(`Authorization`, `^Bearer sky_.+`)
kind: Rule
middlewares:
- name: hm-skypilot-service-account-token-middleware
services:
- name: skypilot-api-service
port: 80
# OAuth2 authentication for browser/other requests
- match: Host(`skypilot.example.com`)
kind: Rule
middlewares:
- name: hm-skypilot-forward-authentication-middleware
services:
- name: skypilot-api-service
port: 80
tls:
secretName: hm-skypilot-certificateRoot Cause
Setting auth.oauth.enabled: true sets the environment variable SKYPILOT_AUTH_OAUTH2_PROXY_ENABLED=true on the API service. This causes SkyPilot to make internal server-side HTTP calls to http://skypilot-oauth2-proxy:4180/oauth2/auth for every request validation.
The problem: these internal server-side calls don't include the browser's _oauth2_proxy session cookie, so OAuth2 proxy returns 401 Unauthorized.
OAuth2 Proxy Logs
"Python/3.10 aiohttp/3.13.2" 401
Every validation call from SkyPilot API fails because the aiohttp request has no session cookie.
Expected Behavior
When using an external ingress controller (like Traefik) with ForwardAuth, SkyPilot should trust the X-Auth-Request-Email header passed by the ingress without making additional internal calls to OAuth2 proxy.
Feature Request
Add a configuration option to trust ingress-level ForwardAuth headers without internal re-validation. For example:
auth:
oauth:
enabled: true
trust-forward-headers: true # Skip internal /oauth2/auth calls, trust X-Auth-Request-Email from ingressOr it would be great to provide documentation for using OAuth2 proxy with non-nginx ingress controllers. Thank you! ☺️
I am hoping to use OAuth2 with existing Traefik in our Kubernetes cluster.
Environment
Problem
I am hoping to use Traefik instead of nginx-ingress. However,
will create
nginx-ingress, so I have to use:The issue is when using
auth.oauth.enabled: truewith Traefik instead of nginx-ingress, users are still prompted for username/password after successfully completing Google OAuth sign-in.Configuration
Helm chart values.yaml:
Traefik ForwardAuth Middleware:
Traefik IngressRoute:
Root Cause
Setting
auth.oauth.enabled: truesets the environment variableSKYPILOT_AUTH_OAUTH2_PROXY_ENABLED=trueon the API service. This causes SkyPilot to make internal server-side HTTP calls tohttp://skypilot-oauth2-proxy:4180/oauth2/authfor every request validation.The problem: these internal server-side calls don't include the browser's
_oauth2_proxysession cookie, so OAuth2 proxy returns 401 Unauthorized.OAuth2 Proxy Logs
Every validation call from SkyPilot API fails because the aiohttp request has no session cookie.
Expected Behavior
When using an external ingress controller (like Traefik) with ForwardAuth, SkyPilot should trust the
X-Auth-Request-Emailheader passed by the ingress without making additional internal calls to OAuth2 proxy.Feature Request
Add a configuration option to trust ingress-level ForwardAuth headers without internal re-validation. For example:
Or it would be great to provide documentation for using OAuth2 proxy with non-nginx ingress controllers. Thank you!☺️