[Pools] Add Support for Autoscaling by lloyd-brown · Pull Request #8483 · skypilot-org/skypilot

lloyd-brown · 2026-01-05T18:05:52Z

This PR adds support for autoscaling pools (even down to 0). The user can now specify in their yaml the max and min number of workers and the target queue length. We introduce a QueueLengthAutoscaler which scales up when the number of pending jobs is more than the queue length and down when it's less.

We also add logic to this autoscaler to prevent us from scaling down past the number of running jobs so that we don't cancel jobs when autoscaling. The downside of this is that if there is a large backlog of jobs we may fail to scale down since a worker that should be scaled down quickly picks up another job. But fixing this would require us to have a new replica status that says a replica should terminate after job completion and the logic there is hairy and I haven't figured it out yet so this is a first cut.

We added some smoke tests to ensure we scale up when our queue length is lower and to make sure we can scale down to zero.

This also address issue #8395

EDIT: Had some discussion with @cg505 about the current algorithm and discussed some changes we should make here and some future work.

The queue length autoscaler as is will scale rather slowly. We inherit the hysteresis autoscaler which by default scales only once every 300 seconds and the autoscaler only scales up by 1 each period. To fix this we could implement some strategy like scaling up based on some factor of the queue length. Setting this factor is tricky, if you set it too high then you might end up setting the number of workers to the max unless the queue is empty.

Since we expect that the chief concern is good utilization (i.e. not wasting workers that are up) we think the current approach is good enough. One thing we should change before the PR is merged that also pertains to utilization is quickly reducing the number of workers to 0 when there are no pending jobs.

In a subsequent PR we will work on a scaling based approach.

Tested (run the relevant ones):

Code formatting: install pre-commit (auto-check on commit) or bash format.sh
Any manual or new tests for this PR (please specify below)
All smoke tests: /smoke-test (CI) or pytest tests/test_smoke.py (local)
Relevant individual tests: /smoke-test -k test_name (CI) or pytest tests/test_smoke.py::test_name (local)
Backward compatibility: /quicktest-core (CI) or pytest tests/smoke_tests/test_backward_compat.py (local)

lloyd-brown · 2026-01-05T18:06:14Z

/smoke-test -k test_pool_autoscaling --kubernetes

gemini-code-assist · 2026-01-05T18:06:24Z

Summary of Changes

Hello @lloyd-brown, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request introduces a significant enhancement to service pools by enabling queue-length-based autoscaling. This allows service pools to automatically adjust their worker count based on the number of pending jobs, improving resource utilization and responsiveness. The changes involve adding a new autoscaler implementation, extending service configuration options, updating YAML parsing and serialization, and introducing a database utility to monitor job queues.

Highlights

Queue-Length Autoscaling for Service Pools: Implemented a new autoscaling mechanism for service pools that dynamically adjusts the number of workers based on the length of the job queue, enhancing resource utilization and responsiveness.
New QueueLengthAutoscaler Class: Introduced a dedicated autoscaler class that monitors pending jobs and scales workers up or down using a configurable queue length threshold and incorporates hysteresis to prevent rapid scaling decisions.
Configurable Pool Parameters: Extended service specifications to allow min_workers, max_workers, queue_length_threshold, upscale_delay_seconds, and downscale_delay_seconds to be defined for service pools, providing fine-grained control over autoscaling behavior.
YAML Schema and Parsing Updates: Modified the YAML schema and parsing logic to support the new dictionary-based pool configuration, enabling users to define comprehensive autoscaling policies directly in their service YAMLs.
New Database Query for Pending Jobs: Added a utility function get_pending_jobs_count_by_pool to efficiently retrieve the number of pending jobs for a specific pool from the database, which is crucial for the queue-length autoscaler.
Comprehensive Smoke Tests: Added new smoke tests to validate the scale-up, scale-down, and boundary conditions of the queue-length autoscaling feature, ensuring its robustness and correctness.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces a queue-length-based autoscaler for service pools. The changes are extensive, touching state management, autoscaler logic, service specification, configuration schemas, and tests. My review focuses on correctness, maintainability, and consistency. I've identified a few issues: a potential bug in how pending jobs are counted, duplicated code in the new autoscaler, inconsistencies in schema validation, and a minor bug in a test helper function. Overall, the feature is well-implemented with good test coverage, and the suggested changes should improve its robustness and maintainability.

lloyd-brown · 2026-01-06T04:20:06Z

/smoke-test -k test_pool_autoscaling --kubernetes

cg505

Core functionality looks good, but there's a lot of code that I think could be cleaned up, especially the SkyServiceSpec.from_yaml_config which seems way too complicated for what it's doing

lloyd-brown · 2026-01-12T23:48:03Z

/smoke-test -k test_pool_autoscaling --kubernetes

lloyd-brown · 2026-01-13T01:52:24Z

/smoke-test -k test_pool_autoscaling --kubernetes

cg505

Thanks for the fixes! The broad strokes look good now, just a few minor fixes

lloyd-brown · 2026-01-27T22:23:46Z

/smoke-test -k test_pool_autoscaling --kubernetes

lloyd-brown · 2026-01-27T22:40:58Z

/smoke-test -k test_pools --kubernetes

lloyd-brown · 2026-01-28T00:11:56Z

/smoke-test -k test_pools --kubernetes

lloyd-brown · 2026-01-28T01:27:47Z

/smoke-test -k test_update_workers --kubernetes

cg505

Thanks for updating everything! LGTM at this point!

Co-authored-by: Christopher Cooper <christopher@cg505.com>

lloyd-brown · 2026-01-29T01:51:37Z

/smoke-test

lloyd-brown · 2026-01-30T22:04:42Z

/smoke-test -k test_pools --kubernetes

Add documentation for the new autoscaling support in pools (PR #8483). - Add "Autoscaling" section covering min_workers/max_workers YAML syntax - Document all autoscaling options: queue_length_threshold, upscale_delay_seconds, downscale_delay_seconds - Explain scale-to-zero behavior (min_workers: 0) - Document how to combine a fixed baseline with autoscaling bounds - Remove "Autoscaling" from the Coming Soon admonition Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering>

* [Docs] Document pool autoscaling feature Add documentation for the new autoscaling support in pools (PR #8483). - Add "Autoscaling" section covering min_workers/max_workers YAML syntax - Document all autoscaling options: queue_length_threshold, upscale_delay_seconds, downscale_delay_seconds - Explain scale-to-zero behavior (min_workers: 0) - Document how to combine a fixed baseline with autoscaling bounds - Remove "Autoscaling" from the Coming Soon admonition Generated with [Claude Code](https://claude.ai/code) via [Happy](https://happy.engineering) Co-Authored-By: Claude <noreply@anthropic.com> Co-Authored-By: Happy <yesreply@happy.engineering> * Update docs/source/examples/pools.rst Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> * [Docs] Add pool section to YAML spec reference (#8927) [Docs] Add pool section to task YAML spec reference Adds documentation for the pool top-level field and its autoscaling sub-fields (workers, min_workers, max_workers, queue_length_threshold, upscale_delay_seconds, downscale_delay_seconds) to the YAML spec reference, matching the existing SkyServe Service section format. * [Docs] Fix undefined label reference in pool yaml spec Change :ref: target from 'job-pool' to 'pool' to match the label defined in pools.rst, fixing the doc build warning. --------- Co-authored-by: Ubuntu <azureuser@agent-vm-78fa7c50-9021-0.cdqox45qxbquxbrpbcn2wku0eg.bx.internal.cloudapp.net> Co-authored-by: Claude <noreply@anthropic.com> Co-authored-by: Happy <yesreply@happy.engineering> Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

gemini-code-assist Bot reviewed Jan 5, 2026

View reviewed changes

Comment thread sky/jobs/state.py Outdated

Comment thread sky/serve/autoscalers.py

Comment thread sky/utils/schemas.py Outdated

Comment thread tests/smoke_tests/test_pools.py

lloyd-brown changed the title ~~test~~ [Pools] Add Support for Autoscaling Jan 6, 2026

lloyd-brown requested review from cg505 and zpoint January 6, 2026 15:30

lloyd-brown marked this pull request as ready for review January 6, 2026 18:32

lloyd-brown removed the request for review from zpoint January 7, 2026 18:38

cg505 reviewed Jan 9, 2026

View reviewed changes

lloyd-brown force-pushed the pools-autoscaling branch from 8e514fe to 972340e Compare January 12, 2026 23:38

lloyd-brown force-pushed the pools-autoscaling branch from 972340e to 9f1c9c3 Compare January 13, 2026 01:50

lloyd-brown requested a review from cg505 January 13, 2026 02:13

cg505 reviewed Jan 15, 2026

View reviewed changes

Comment thread sky/serve/service_spec.py Outdated

Comment thread sky/serve/service_spec.py Outdated

Comment thread sky/serve/autoscalers.py Outdated

Comment thread sky/serve/autoscalers.py

Comment thread sky/serve/service_spec.py

Comment thread sky/utils/schemas.py Outdated

lloyd-brown requested review from Copilot and removed request for Copilot January 27, 2026 10:46

Copilot started reviewing on behalf of lloyd-brown January 27, 2026 10:46 View session

lloyd-brown added 10 commits January 27, 2026 13:53

Working autoscaling.

bf928cd

forma

c035c25

Fixes.

1a36864

Many fixes.

8121a83

Fix some of the policy.

9dcd0c7

More fixes.

ca2d20e

Simplify logic, add quick downscaling.

3c5846b

More simplification.

806befc

Change some schema validation.

6b987fa

Change spec.

d9eb5b9

lloyd-brown force-pushed the pools-autoscaling branch from 138dd72 to d9eb5b9 Compare January 27, 2026 21:55

lloyd-brown added 2 commits January 27, 2026 14:05

Add unit tests.

cf6e904

Format

4aa0456

lloyd-brown requested a review from cg505 January 27, 2026 22:41

Fix queue length setting.

39fd319

Fix updating.

e587782

lloyd-brown requested a review from cblmemo January 28, 2026 23:43

cg505 approved these changes Jan 29, 2026

View reviewed changes

Comment thread sky/serve/service_spec.py Outdated

Update sky/serve/service_spec.py

df945ce

Co-authored-by: Christopher Cooper <christopher@cg505.com>

cblmemo requested changes Jan 29, 2026

View reviewed changes

Comment thread sky/serve/autoscalers.py

Comment thread sky/serve/autoscalers.py

Comment thread sky/serve/autoscalers.py

Add todo about scaling factor.

c46ac96

lloyd-brown requested a review from cblmemo January 30, 2026 00:09

cblmemo approved these changes Jan 30, 2026

View reviewed changes

Fix min replicas.

da6cb61

lloyd-brown merged commit 71c5163 into master Jan 30, 2026
22 checks passed

lloyd-brown deleted the pools-autoscaling branch January 30, 2026 22:44

kevinmingtarja mentioned this pull request Feb 11, 2026

[Docs] Update pools page to document shipped features #8829

Closed

lloyd-brown mentioned this pull request Feb 18, 2026

[Docs] Document pool autoscaling feature #8873

Merged

Uh oh!

Conversation

lloyd-brown commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lloyd-brown commented Jan 5, 2026

Uh oh!

gemini-code-assist Bot commented Jan 5, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lloyd-brown commented Jan 6, 2026

Uh oh!

cg505 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lloyd-brown commented Jan 12, 2026

Uh oh!

lloyd-brown commented Jan 13, 2026

Uh oh!

cg505 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lloyd-brown commented Jan 27, 2026

Uh oh!

lloyd-brown commented Jan 27, 2026

Uh oh!

lloyd-brown commented Jan 28, 2026

Uh oh!

lloyd-brown commented Jan 28, 2026

Uh oh!

cg505 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

lloyd-brown commented Jan 29, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lloyd-brown commented Jan 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lloyd-brown commented Jan 5, 2026 •

edited

Loading