perf(tsdb): inline oversize-chunk check in populateChunksFromIterable by aknuds1 · Pull Request #18699 · prometheus/prometheus

aknuds1 · 2026-05-15T09:39:21Z

Which issue(s) does the PR fix:

The oversize-chunk trigger introduced in #18692 was implemented as a closure defined inside the per-sample loop in
populateChunksFromIterable and invoked once at the if condition. Replace it with a plain conditional and hoist
len(currentChunk.Bytes()) out of the switch so the two encoding cases don't repeat the same expression. The new shape preserves the original || short-circuit: the size check is only evaluated when neither the encoding nor the start-timestamp capability forces a new chunk, which also keeps currentChunk non-nil at the point of read.

gcflags=-m=2 reports the closure body inlined and the symbol table shows no separate func1 symbol, yet benchstat shows a measurable speedup. The most likely explanation: the closure body inlines, but the funcval struct (capturing currentChunk and currentValueType) is still stack-constructed each iteration — invisible to escape analysis, but a real per-iteration cost in a hot loop.

Benchmark, go test -count=6 -benchmem -bench=BenchmarkQuerierSelectWithOutOfOrder -benchtime=5s -run=^$ ./tsdb/, Intel Xeon Platinum 8280 @ 2.70 GHz (linux/amd64), 1M-series head, query selectivity varies:

                                          │   main      │           optimized                │
                                          │   sec/op    │    sec/op    vs base               │
Head/1of1000000-16                          301.5m ± 4%   257.0m ±  4%  -14.74% (p=0.002 n=6)
Head/10of1000000-16                         305.6m ± 3%   260.4m ±  2%  -14.80% (p=0.002 n=6)
Head/100of1000000-16                        303.9m ± 2%   259.7m ±  2%  -14.54% (p=0.002 n=6)
Head/1000of1000000-16                       303.8m ± 2%   267.0m ±  2%  -12.13% (p=0.002 n=6)
Head/10000of1000000-16                      318.1m ± 1%   278.9m ±  8%  -12.33% (p=0.002 n=6)
Head/100000of1000000-16                     364.1m ± 7%   352.8m ±  4%        ~ (p=0.065 n=6)
Head/1000000of1000000-16                     1.115 ± 2%    1.089 ± 26%        ~ (p=0.394 n=6)
geomean                                     377.8m        337.3m        -10.71%

allocs/op and B/op unchanged. The two largest-selectivity cases trend faster but are dominated by the per-sample append cost so the relative delta is smaller and lost in variance.

TestChunkQuerier_OverlappingInOrderAndOOOChunks continues to exercise the overflow path.

Release notes for end users (ALL commits must be considered).

Reviewers should verify clarity and quality.

NONE

aknuds1 · 2026-05-15T09:39:43Z

~~I'm going to confirm the benchmark stats in a cloud VM.~~ Done.

The oversize-chunk trigger introduced in prometheus#18692 was implemented as a closure defined inside the per-sample loop in populateChunksFromIterable and invoked once at the `if` condition. Replace it with a plain conditional and hoist `len(currentChunk.Bytes())` out of the switch so the two encoding cases don't repeat the same expression. The new shape preserves the original `||` short-circuit: the size check is only evaluated when neither the encoding nor the start-timestamp capability forces a new chunk, which also keeps `currentChunk` non-nil at the point of read. `gcflags=-m=2` reports the closure body inlined and the symbol table shows no separate `func1` symbol, yet benchstat shows a measurable speedup. The most likely explanation: the closure body inlines, but the `funcval` struct (capturing `currentChunk` and `currentValueType`) is still stack-constructed each iteration — invisible to escape analysis, but a real per-iteration cost in a hot loop. Benchmark, `go test -count=6 -benchmem -bench=BenchmarkQuerierSelectWithOutOfOrder -benchtime=5s -run=^$ ./tsdb/`, Intel Xeon Platinum 8280 @ 2.70 GHz (linux/amd64), 1M-series head, query selectivity varies: │ main │ optimized │ │ sec/op │ sec/op vs base │ Head/1of1000000-16 301.5m ± 4% 257.0m ± 4% -14.74% (p=0.002 n=6) Head/10of1000000-16 305.6m ± 3% 260.4m ± 2% -14.80% (p=0.002 n=6) Head/100of1000000-16 303.9m ± 2% 259.7m ± 2% -14.54% (p=0.002 n=6) Head/1000of1000000-16 303.8m ± 2% 267.0m ± 2% -12.13% (p=0.002 n=6) Head/10000of1000000-16 318.1m ± 1% 278.9m ± 8% -12.33% (p=0.002 n=6) Head/100000of1000000-16 364.1m ± 7% 352.8m ± 4% ~ (p=0.065 n=6) Head/1000000of1000000-16 1.115 ± 2% 1.089 ± 26% ~ (p=0.394 n=6) geomean 377.8m 337.3m -10.71% allocs/op and B/op unchanged. The two largest-selectivity cases trend faster but are dominated by the per-sample append cost so the relative delta is smaller and lost in variance. `TestChunkQuerier_OverlappingInOrderAndOOOChunks` continues to exercise the overflow path. ```release-notes NONE ``` Signed-off-by: Arve Knudsen <arve.knudsen@gmail.com>

krajorama

LGTM

…➔ v3.13.0) (#1360) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [quay.io/prometheus/prometheus](https://github.com/prometheus/prometheus) | minor | `v3.12.0` → `v3.13.0` | --- ### Release Notes <details> <summary>prometheus/prometheus (quay.io/prometheus/prometheus)</summary> ### [`v3.13.0`](https://github.com/prometheus/prometheus/releases/tag/v3.13.0): 3.13.0 / 2026-07-01 [Compare Source](prometheus/prometheus@v3.12.0...v3.13.0) This is a Long Term Support [LTS](https://prometheus.io/docs/introduction/release-cycle/) release. - \[SECURITY] UI: Bump `sanitize-html` to fix a cross-site scripting vulnerability (CVE-2026-44990). [#18697](prometheus/prometheus#18697) - \[CHANGE] UI: Third-party npm dependency licenses are now embedded in the Prometheus binary and served at `/assets/third-party-licenses.txt`, replacing the `npm_licenses.tar.bz2` archive previously shipped in release tarballs and container images. [#18997](prometheus/prometheus#18997) - \[CHANGE] API: Use SHA-256 instead of SHA-1 to generate rule group pagination tokens. [#18927](prometheus/prometheus#18927) - \[CHANGE] HTTP clients: Credentials (Authorization header, basic auth, bearer token, OAuth2, configured headers) are no longer forwarded when following a redirect to a different host; affects scraping, remote read/write, alerting, and service discovery. Via prometheus/common v0.69.0 (CVE-2025-4673 CVE-2023-45289). [#18949](prometheus/prometheus#18949) - \[CHANGE] promtool: Relative file paths in the file passed to `--http.config.file` are now resolved relative to that config file's directory instead of its parent directory. Via prometheus/common v0.69.0. [#18949](prometheus/prometheus#18949) - \[CHANGE] PromQL: Rename the `min()` and `max()` duration-expression functions (experimental feature flag `experimental-duration-expr`) to `min_of()` and `max_of()` to avoid confusion with the `min` and `max` aggregate operators. [#18687](prometheus/prometheus#18687) - \[FEATURE] API: Add experimental search endpoints to search metric names, label names, and label values. [#18573](prometheus/prometheus#18573) - \[FEATURE] Discovery/AWS: Add ability to filter RDS instances. [#18859](prometheus/prometheus#18859) - \[FEATURE] PromQL: Add `min_of(a, b)` and `max_of(a, b)` scalar experimental functions, returning the smaller or larger of two scalar values. [#18687](prometheus/prometheus#18687) - \[FEATURE] PromQL: Add support for smoothed/anchored rate with native histograms. [#18564](prometheus/prometheus#18564) - \[FEATURE] PromQL: Expose per-query `samplesRead` (and `samplesReadPerStep` with `stats=all` and the `promql-per-step-stats` feature flag) in the query stats response, and add the `prometheus_engine_query_samples_read_total` engine counter. `samplesRead` reflects storage I/O distinct from `totalQueryableSamples`, which counts samples loaded into the evaluator (and so over-counts when a sample is reused across multiple range-vector windows). [#18081](prometheus/prometheus#18081) - \[FEATURE] Scrape: Add `__convert_classic_histograms_to_nhcb__` internal label to allow per-target override of `convert_classic_histograms_to_nhcb` scrape configuration via relabeling. [#18840](prometheus/prometheus#18840) - \[FEATURE] TSDB: Add `storage.tsdb.chunk_encoding.floats` configuration field to select float chunk encoding (`xor` or `xor2`) at runtime, independently of the `--enable-feature=xor2-encoding` flag. [#18769](prometheus/prometheus#18769) - \[FEATURE] remote\_write: Add Certificate support for ingesting data into an Azure Monitor Workspace. [#18217](prometheus/prometheus#18217) - \[FEATURE] Scrape: Add `__always_scrape_classic_histograms__` and `__scrape_native_histograms__` internal labels to allow per-target override of the `always_scrape_classic_histograms` and `scrape_native_histograms` scrape configuration via relabeling. [#18929](prometheus/prometheus#18929) - \[ENHANCEMENT] Release: Container images are now also published to the GitHub Container Registry (ghcr.io). [#18791](prometheus/prometheus#18791) - \[ENHANCEMENT] PromQL: Prettify `fill_left(x) fill_right(x)` as `fill(x)` when both fill values are equal. [#18851](prometheus/prometheus#18851) - \[ENHANCEMENT] UI: Improve autocompletion after closing a function bracket. [#18894](prometheus/prometheus#18894) - \[PERF] Labels: Add case-insensitive prefix matching to speed up evaluation of long case-insensitive regular expressions (up to \~2x faster). [#18540](prometheus/prometheus#18540) - \[PERF] TSDB: Reduce per-sample overhead in chunk population, speeding up affected queries by \~12-15% in benchmarks. [#18699](prometheus/prometheus#18699) - \[PERF] TSDB: Eliminate unnecessary heap allocations in the V2 histogram WAL decoder, reducing allocations by up to 50% and memory by up to 10% for deployments using native histograms with created-timestamp storage enabled (`--enable-feature=created-timestamp-zero-ingestion`). [#18813](prometheus/prometheus#18813) - \[BUGFIX] Discovery/AWS: Fix failure when processing an AWS RDS cluster without instances. [#18845](prometheus/prometheus#18845) - \[BUGFIX] Fix race condition in initTime that could cause ErrOutOfBounds. [#18629](prometheus/prometheus#18629) - \[BUGFIX] PromQL: A range query whose `end` was not aligned to `step` caused subqueries inside it to evaluate past the parent's last actual step, inflating `peakSamples` in the query stats and against the `query.max-samples` limit, and wasting storage I/O reading samples that were never used in the result. [#18081](prometheus/prometheus#18081) - \[BUGFIX] PromQL: A range query containing an at-modifier-unsafe function over a range-vector with an `@` modifier (e.g. `predict_linear(metric[60s] @ T, X)`) silently under-counted `totalQueryableSamples` for steps after step 0. [#18081](prometheus/prometheus#18081) - \[BUGFIX] PromQL: Fix `fill_left`/`fill_right` producing missing samples in range queries when using `group_left`/`group_right`. [#18850](prometheus/prometheus#18850) - \[BUGFIX] PromQL: Fix for resets() and changes() in anchored range extenders with histograms. [#18906](prometheus/prometheus#18906) - \[BUGFIX] PromQL: Fix panic on `1[5m] smoothed` and similar expressions when extended range selectors are enabled. [#18764](prometheus/prometheus#18764) - \[BUGFIX] PromQL: Fix panic when a `smoothed` instant vector selector produces no samples for a series. [#18943](prometheus/prometheus#18943) - \[BUGFIX] PromQL: Fix panic when using a parenthesised plain number as an offset (e.g. `foo offset -(5)`). [#18768](prometheus/prometheus#18768) - \[BUGFIX] promtool: Fix panic when parsing exposition text containing empty braces `{}`. Via prometheus/common v0.69.0. [#18949](prometheus/prometheus#18949) - \[BUGFIX] Promtool: Fix `check healthy` and `check ready` when `--url` ends with a trailing slash. [#18854](prometheus/prometheus#18854) - \[BUGFIX] Rules: Close PromQL query after each rule evaluation to ensure resources are released. [#18733](prometheus/prometheus#18733) - \[BUGFIX] Scaleway SD: Resolve VPC/IPAM-only instances that have no legacy `private_ip` or `public_ip` field, but do have private NICs attached. [#18772](prometheus/prometheus#18772) - \[BUGFIX] TSDB: Do not leak head series when an integer histogram append is rejected (e.g. out-of-order). [#18838](prometheus/prometheus#18838) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] TSDB: Fix chunk snapshot encoding for EncXOR2 chunks, preventing corruption on TSDB restart when EncXOR2-encoded series were present. [#18739](prometheus/prometheus#18739) - \[BUGFIX] TSDB: Store a millisecond timestamp (not a WAL segment number) in walExpiries when a series is evicted via CompactStaleHead/CompactSelectedSeries, so the series's label record is correctly retained in the next WAL checkpoint and replays cleanly. [#18847](prometheus/prometheus#18847) - \[BUGFIX] TSDB: Prevent loss of samples at the chunk-range boundary when CompactSelectedSeries (and CompactStaleHead) evict the series — the per-slice compaction loop now runs one more iteration so the boundary timestamp is captured in a block before the in-memory copy is removed. [#18849](prometheus/prometheus#18849) </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.erwanleboucher.dev/eleboucher/homelab/pulls/1360

##### [\`v3.13.0\`](https://github.com/prometheus/prometheus/releases/tag/v3.13.0) This is a Long Term Support [LTS](https://prometheus.io/docs/introduction/release-cycle/) release. - \[SECURITY] UI: Bump `sanitize-html` to fix a cross-site scripting vulnerability (CVE-2026-44990). [#18697](prometheus/prometheus#18697) - \[CHANGE] UI: Third-party npm dependency licenses are now embedded in the Prometheus binary and served at `/assets/third-party-licenses.txt`, replacing the `npm_licenses.tar.bz2` archive previously shipped in release tarballs and container images. [#18997](prometheus/prometheus#18997) - \[CHANGE] API: Use SHA-256 instead of SHA-1 to generate rule group pagination tokens. [#18927](prometheus/prometheus#18927) - \[CHANGE] HTTP clients: Credentials (Authorization header, basic auth, bearer token, OAuth2, configured headers) are no longer forwarded when following a redirect to a different host; affects scraping, remote read/write, alerting, and service discovery. Via prometheus/common v0.69.0 (CVE-2025-4673 CVE-2023-45289). [#18949](prometheus/prometheus#18949) - \[CHANGE] promtool: Relative file paths in the file passed to `--http.config.file` are now resolved relative to that config file's directory instead of its parent directory. Via prometheus/common v0.69.0. [#18949](prometheus/prometheus#18949) - \[CHANGE] PromQL: Rename the `min()` and `max()` duration-expression functions (experimental feature flag `experimental-duration-expr`) to `min_of()` and `max_of()` to avoid confusion with the `min` and `max` aggregate operators. [#18687](prometheus/prometheus#18687) - \[FEATURE] API: Add experimental search endpoints to search metric names, label names, and label values. [#18573](prometheus/prometheus#18573) - \[FEATURE] Discovery/AWS: Add ability to filter RDS instances. [#18859](prometheus/prometheus#18859) - \[FEATURE] PromQL: Add `min_of(a, b)` and `max_of(a, b)` scalar experimental functions, returning the smaller or larger of two scalar values. [#18687](prometheus/prometheus#18687) - \[FEATURE] PromQL: Add support for smoothed/anchored rate with native histograms. [#18564](prometheus/prometheus#18564) - \[FEATURE] PromQL: Expose per-query `samplesRead` (and `samplesReadPerStep` with `stats=all` and the `promql-per-step-stats` feature flag) in the query stats response, and add the `prometheus_engine_query_samples_read_total` engine counter. `samplesRead` reflects storage I/O distinct from `totalQueryableSamples`, which counts samples loaded into the evaluator (and so over-counts when a sample is reused across multiple range-vector windows). [#18081](prometheus/prometheus#18081) - \[FEATURE] Scrape: Add `__convert_classic_histograms_to_nhcb__` internal label to allow per-target override of `convert_classic_histograms_to_nhcb` scrape configuration via relabeling. [#18840](prometheus/prometheus#18840) - \[FEATURE] TSDB: Add `storage.tsdb.chunk_encoding.floats` configuration field to select float chunk encoding (`xor` or `xor2`) at runtime, independently of the `--enable-feature=xor2-encoding` flag. [#18769](prometheus/prometheus#18769) - \[FEATURE] remote\_write: Add Certificate support for ingesting data into an Azure Monitor Workspace. [#18217](prometheus/prometheus#18217) - \[FEATURE] Scrape: Add `__always_scrape_classic_histograms__` and `__scrape_native_histograms__` internal labels to allow per-target override of the `always_scrape_classic_histograms` and `scrape_native_histograms` scrape configuration via relabeling. [#18929](prometheus/prometheus#18929) - \[ENHANCEMENT] Release: Container images are now also published to the GitHub Container Registry (ghcr.io). [#18791](prometheus/prometheus#18791) - \[ENHANCEMENT] PromQL: Prettify `fill_left(x) fill_right(x)` as `fill(x)` when both fill values are equal. [#18851](prometheus/prometheus#18851) - \[ENHANCEMENT] UI: Improve autocompletion after closing a function bracket. [#18894](prometheus/prometheus#18894) - \[PERF] Labels: Add case-insensitive prefix matching to speed up evaluation of long case-insensitive regular expressions (up to \~2x faster). [#18540](prometheus/prometheus#18540) - \[PERF] TSDB: Reduce per-sample overhead in chunk population, speeding up affected queries by \~12-15% in benchmarks. [#18699](prometheus/prometheus#18699) - \[PERF] TSDB: Eliminate unnecessary heap allocations in the V2 histogram WAL decoder, reducing allocations by up to 50% and memory by up to 10% for deployments using native histograms with created-timestamp storage enabled (`--enable-feature=created-timestamp-zero-ingestion`). [#18813](prometheus/prometheus#18813) - \[BUGFIX] Discovery/AWS: Fix failure when processing an AWS RDS cluster without instances. [#18845](prometheus/prometheus#18845) - \[BUGFIX] Fix race condition in initTime that could cause ErrOutOfBounds. [#18629](prometheus/prometheus#18629) - \[BUGFIX] PromQL: A range query whose `end` was not aligned to `step` caused subqueries inside it to evaluate past the parent's last actual step, inflating `peakSamples` in the query stats and against the `query.max-samples` limit, and wasting storage I/O reading samples that were never used in the result. [#18081](prometheus/prometheus#18081) - \[BUGFIX] PromQL: A range query containing an at-modifier-unsafe function over a range-vector with an `@` modifier (e.g. `predict_linear(metric[60s] @ T, X)`) silently under-counted `totalQueryableSamples` for steps after step 0. [#18081](prometheus/prometheus#18081) - \[BUGFIX] PromQL: Fix `fill_left`/`fill_right` producing missing samples in range queries when using `group_left`/`group_right`. [#18850](prometheus/prometheus#18850) - \[BUGFIX] PromQL: Fix for resets() and changes() in anchored range extenders with histograms. [#18906](prometheus/prometheus#18906) - \[BUGFIX] PromQL: Fix panic on `1[5m] smoothed` and similar expressions when extended range selectors are enabled. [#18764](prometheus/prometheus#18764) - \[BUGFIX] PromQL: Fix panic when a `smoothed` instant vector selector produces no samples for a series. [#18943](prometheus/prometheus#18943) - \[BUGFIX] PromQL: Fix panic when using a parenthesised plain number as an offset (e.g. `foo offset -(5)`). [#18768](prometheus/prometheus#18768) - \[BUGFIX] promtool: Fix panic when parsing exposition text containing empty braces `{}`. Via prometheus/common v0.69.0. [#18949](prometheus/prometheus#18949) - \[BUGFIX] Promtool: Fix `check healthy` and `check ready` when `--url` ends with a trailing slash. [#18854](prometheus/prometheus#18854) - \[BUGFIX] Rules: Close PromQL query after each rule evaluation to ensure resources are released. [#18733](prometheus/prometheus#18733) - \[BUGFIX] Scaleway SD: Resolve VPC/IPAM-only instances that have no legacy `private_ip` or `public_ip` field, but do have private NICs attached. [#18772](prometheus/prometheus#18772) - \[BUGFIX] TSDB: Do not leak head series when an integer histogram append is rejected (e.g. out-of-order). [#18838](prometheus/prometheus#18838) - \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#18658](prometheus/prometheus#18658) - \[BUGFIX] TSDB: Fix chunk snapshot encoding for EncXOR2 chunks, preventing corruption on TSDB restart when EncXOR2-encoded series were present. [#18739](prometheus/prometheus#18739) - \[BUGFIX] TSDB: Store a millisecond timestamp (not a WAL segment number) in walExpiries when a series is evicted via CompactStaleHead/CompactSelectedSeries, so the series's label record is correctly retained in the next WAL checkpoint and replays cleanly. [#18847](prometheus/prometheus#18847) - \[BUGFIX] TSDB: Prevent loss of samples at the chunk-range boundary when CompactSelectedSeries (and CompactStaleHead) evict the series — the per-slice compaction loop now runs one more iteration so the boundary timestamp is captured in a block before the in-memory copy is removed. [#18849](prometheus/prometheus#18849)

aknuds1 force-pushed the arve/fix-promql-closure branch from 78e8f6c to 6c439c6 Compare May 15, 2026 10:32

aknuds1 marked this pull request as ready for review May 15, 2026 10:37

aknuds1 requested review from a team, bwplotka, codesome, jesusvazquez and krajorama as code owners May 15, 2026 10:37

aknuds1 added the kind/performance label May 15, 2026

aknuds1 added the component/tsdb label May 15, 2026

aknuds1 force-pushed the arve/fix-promql-closure branch from 6c439c6 to ead94d8 Compare May 15, 2026 10:39

krajorama approved these changes May 15, 2026

View reviewed changes

aknuds1 enabled auto-merge (squash) May 15, 2026 10:44

aknuds1 merged commit 43e5fc6 into prometheus:main May 15, 2026
57 of 58 checks passed

aknuds1 deleted the arve/fix-promql-closure branch May 15, 2026 11:45

krajorama mentioned this pull request May 15, 2026

update from upstream main at 43e5fc6a248c1a73d30dff86428f74d11aedbedc grafana/mimir-prometheus#1162

Merged

krajorama mentioned this pull request Jun 18, 2026

chore: prepare 3.13.0-rc.0 release #18965

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

perf(tsdb): inline oversize-chunk check in populateChunksFromIterable#18699

perf(tsdb): inline oversize-chunk check in populateChunksFromIterable#18699
aknuds1 merged 1 commit into
prometheus:mainfrom
aknuds1:arve/fix-promql-closure

aknuds1 commented May 15, 2026 •

edited

Loading

Uh oh!

aknuds1 commented May 15, 2026 •

edited

Loading

Uh oh!

krajorama left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

aknuds1 commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which issue(s) does the PR fix:

Release notes for end users (ALL commits must be considered).

Uh oh!

aknuds1 commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

krajorama left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aknuds1 commented May 15, 2026 •

edited

Loading

aknuds1 commented May 15, 2026 •

edited

Loading