Skip to content

web/api: add search API endpoint #18573

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
roidelapluie merged 5 commits into from
May 19, 2026
Merged

web/api: add search API endpoint #18573

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
e8e25eb
web/api: add search API endpoint
roidelapluie Apr 23, 2026
99ef765
chore: fix build
roidelapluie May 18, 2026
d82cf5f
Merge remote-tracking branch 'origin/main' into roidelapluie/search-api
roidelapluie May 18, 2026
e713c64
chore: fix go-fmt and tests
roidelapluie May 18, 2026
9170a8e
feat(search): rename `name` field to `value` in label_values search r…
roidelapluie May 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 12 additions & 1 deletion cmd/prometheus/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -337,6 +337,9 @@ func (c *flagConfig) setFeatureListOptions(logger *slog.Logger) error {
case "fast-startup":
c.tsdb.EnableFastStartup = true
logger.Info("Experimental fast startup is enabled.")
case "search-api":
c.web.EnableSearch = true
logger.Info("Experimental search API enabled.")
default:
logger.Warn("Unknown option for --enable-feature", "option", o)
}
Expand Down Expand Up @@ -583,6 +586,9 @@ func main() {
serverOnlyFlag(a, "storage.remote.read-max-bytes-in-frame", "Maximum number of bytes in a single frame for streaming remote read response types before marshalling. Note that client might have limit on frame size as well. 1MB as recommended by protobuf by default.").
Default("1048576").IntVar(&cfg.web.RemoteReadBytesInFrame)

serverOnlyFlag(a, "web.search.max-limit", "Hard upper bound on the \"limit\" query parameter accepted by the experimental search API (--enable-feature=search-api). Requests with a higher limit are rejected with HTTP 400. 0 disables the cap.").
Default("10000").IntVar(&cfg.web.MaxSearchLimit)

serverOnlyFlag(a, "rules.alert.for-outage-tolerance", "Max time to tolerate prometheus outage for restoring \"for\" state of alert.").
Default("1h").SetValue(&cfg.outageTolerance)

Expand Down Expand Up @@ -625,7 +631,7 @@ func main() {
a.Flag("scrape.discovery-reload-interval", "Interval used by scrape manager to throttle target groups updates.").
Hidden().Default("5s").SetValue(&cfg.scrape.DiscoveryReloadInterval)

a.Flag("enable-feature", "Comma separated feature names to enable. Valid options: concurrent-rule-eval, created-timestamp-zero-ingestion, delayed-compaction, exemplar-storage, extra-scrape-metrics, memory-snapshot-on-shutdown, metadata-wal-records, old-ui, otlp-deltatocumulative, otlp-native-delta-ingestion, promql-binop-fill-modifiers, promql-delayed-name-removal, promql-duration-expr, promql-experimental-functions, promql-extended-range-selectors, promql-per-step-stats, st-storage, st-synthesis, type-and-unit-labels, use-start-timestamps, use-uncached-io, xor2-encoding. See https://prometheus.io/docs/prometheus/latest/feature_flags/ for more details.").
a.Flag("enable-feature", "Comma separated feature names to enable. Valid options: concurrent-rule-eval, created-timestamp-zero-ingestion, delayed-compaction, exemplar-storage, extra-scrape-metrics, memory-snapshot-on-shutdown, metadata-wal-records, old-ui, otlp-deltatocumulative, otlp-native-delta-ingestion, promql-binop-fill-modifiers, promql-delayed-name-removal, promql-duration-expr, promql-experimental-functions, promql-extended-range-selectors, promql-per-step-stats, search-api, st-storage, st-synthesis, type-and-unit-labels, use-start-timestamps, use-uncached-io, xor2-encoding. See https://prometheus.io/docs/prometheus/latest/feature_flags/ for more details.").
StringsVar(&cfg.featureList)

a.Flag("agent", "Run Prometheus in 'Agent mode'.").BoolVar(&agentMode)
Expand Down Expand Up @@ -668,6 +674,11 @@ func main() {
logger.Info("Automatic configuration file reloading enabled", "interval", cfg.autoReloadInterval)
}

if cfg.web.MaxSearchLimit < 0 {
fmt.Fprintf(os.Stderr, "--web.search.max-limit must be non-negative; got %d (use 0 to disable the cap)\n", cfg.web.MaxSearchLimit)
os.Exit(1)
}

promqlParser := parser.NewParser(cfg.parserOpts)

if agentMode && len(serverOnlyFlags) > 0 {
Expand Down
3 changes: 3 additions & 0 deletions cmd/prometheus/testdata/features.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@
"query_stats": true,
"query_warnings": true,
"remote_write_receiver": false,
"search": false,
"search_fuzz_alg_jarowinkler": true,
"search_fuzz_alg_subsequence": true,
"time_range_labels": true,
"time_range_series": true
},
Expand Down
3 changes: 2 additions & 1 deletion docs/command-line/prometheus.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ The Prometheus monitoring server
| <code class="text-nowrap">--storage.remote.read-sample-limit</code> | Maximum overall number of samples to return via the remote read interface, in a single query. 0 means no limit. This limit is ignored for streamed response types. Use with server mode only. | `5e7` |
| <code class="text-nowrap">--storage.remote.read-concurrent-limit</code> | Maximum number of concurrent remote read calls. 0 means no limit. Use with server mode only. | `10` |
| <code class="text-nowrap">--storage.remote.read-max-bytes-in-frame</code> | Maximum number of bytes in a single frame for streaming remote read response types before marshalling. Note that client might have limit on frame size as well. 1MB as recommended by protobuf by default. Use with server mode only. | `1048576` |
| <code class="text-nowrap">--web.search.max-limit</code> | Hard upper bound on the "limit" query parameter accepted by the experimental search API (--enable-feature=search-api). Requests with a higher limit are rejected with HTTP 400. 0 disables the cap. Use with server mode only. | `10000` |
| <code class="text-nowrap">--rules.alert.for-outage-tolerance</code> | Max time to tolerate prometheus outage for restoring "for" state of alert. Use with server mode only. | `1h` |
| <code class="text-nowrap">--rules.alert.for-grace-period</code> | Minimum duration between alert and restored "for" state. This is maintained only for alerts with configured "for" time greater than grace period. Use with server mode only. | `10m` |
| <code class="text-nowrap">--rules.alert.resend-delay</code> | Minimum amount of time to wait before resending an alert to Alertmanager. Use with server mode only. | `1m` |
Expand All @@ -62,7 +63,7 @@ The Prometheus monitoring server
| <code class="text-nowrap">--query.timeout</code> | Maximum time a query may take before being aborted. Use with server mode only. | `2m` |
| <code class="text-nowrap">--query.max-concurrency</code> | Maximum number of queries executed concurrently. Use with server mode only. | `20` |
| <code class="text-nowrap">--query.max-samples</code> | Maximum number of samples a single query can load into memory. Note that queries will fail if they try to load more samples than this into memory, so this also limits the number of samples a query can return. Use with server mode only. | `50000000` |
| <code class="text-nowrap">--enable-feature</code> <code class="text-nowrap">...<code class="text-nowrap"> | Comma separated feature names to enable. Valid options: concurrent-rule-eval, created-timestamp-zero-ingestion, delayed-compaction, exemplar-storage, extra-scrape-metrics, memory-snapshot-on-shutdown, metadata-wal-records, old-ui, otlp-deltatocumulative, otlp-native-delta-ingestion, promql-binop-fill-modifiers, promql-delayed-name-removal, promql-duration-expr, promql-experimental-functions, promql-extended-range-selectors, promql-per-step-stats, st-storage, st-synthesis, type-and-unit-labels, use-start-timestamps, use-uncached-io, xor2-encoding. See https://prometheus.io/docs/prometheus/latest/feature_flags/ for more details. | |
| <code class="text-nowrap">--enable-feature</code> <code class="text-nowrap">...<code class="text-nowrap"> | Comma separated feature names to enable. Valid options: concurrent-rule-eval, created-timestamp-zero-ingestion, delayed-compaction, exemplar-storage, extra-scrape-metrics, memory-snapshot-on-shutdown, metadata-wal-records, old-ui, otlp-deltatocumulative, otlp-native-delta-ingestion, promql-binop-fill-modifiers, promql-delayed-name-removal, promql-duration-expr, promql-experimental-functions, promql-extended-range-selectors, promql-per-step-stats, search-api, st-storage, st-synthesis, type-and-unit-labels, use-start-timestamps, use-uncached-io, xor2-encoding. See https://prometheus.io/docs/prometheus/latest/feature_flags/ for more details. | |
| <code class="text-nowrap">--agent</code> | Run Prometheus in 'Agent mode'. | |
| <code class="text-nowrap">--log.level</code> | Only log messages with the given severity or above. One of: [debug, info, warn, error] | `info` |
| <code class="text-nowrap">--log.format</code> | Output format of log messages. One of: [logfmt, json] | `logfmt` |
Expand Down
18 changes: 18 additions & 0 deletions docs/feature_flags.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -401,3 +401,21 @@ Example query:
```

See [the fill modifiers documentation](querying/operators.md#filling-in-missing-matches) for more details and examples.


## Search API

`--enable-feature=search-api`

Enables the experimental search API endpoints for discovering metric names,
label names, and label values with fuzzy matching and filtering support. See
[the search API documentation](querying/api.md#searching-metric-names-label-names-and-label-values)
for details.

The `--web.search.max-limit` flag (default `10000`) bounds the `limit` query
parameter accepted by the search endpoints. Requests with a higher `limit` are
rejected with HTTP 400. The default response limit (100) is silently clamped
to this maximum, so an operator setting a smaller cap does not break
no-`limit` requests. Setting the flag to `0` disables the cap entirely; this
is **not recommended** for endpoints exposed beyond a trusted network because a
single client can then request the entire index in one response.
108 changes: 108 additions & 0 deletions docs/querying/api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -521,6 +521,114 @@ curl http://localhost:9090/api/v1/label/U__http_2e_status_code/values
}
```

### Searching metric names, label names, and label values

These endpoints are experimental and must be enabled with
`--enable-feature=search-api`.

The following endpoints provide streamed discovery results for metric names,
label names, and label values:

```
GET /api/v1/search/metric_names
POST /api/v1/search/metric_names
GET /api/v1/search/label_names
POST /api/v1/search/label_names
GET /api/v1/search/label_values
POST /api/v1/search/label_values
```

These endpoints return newline-delimited JSON with content type
`application/x-ndjson`. The stream contract is:

- Zero or more **batch lines**, each with a `results` array and an optional
`warnings` array.
- The stream then ends with **either** a **trailer line** (`status`, `has_more`,
optional `warnings`) **or** an **error line** (`status`, `errorType`, `error`)
if iteration failed mid-stream after the first batch was sent.

```json
{"results":[{"name":"http_requests_total","type":"counter","help":"Total HTTP requests."}]}
{"status":"success","has_more":false}
```

If an error occurs **before** streaming starts, the API returns the usual
Prometheus JSON error object with a 4xx/5xx status code. If an error occurs
**after** streaming starts, the stream ends with an NDJSON error line in place
of the trailer.

Clients must tolerate an abrupt EOF without a trailer (for example, on
transport failure or server shutdown) and must ignore unknown fields in the
trailer for forward compatibility.

The `has_more` field in the trailer is informational only: this version of
the API does not provide a pagination cursor. To retrieve more results, raise
`limit` (subject to the operator-configured `--web.search.max-limit`) or narrow
the request via `match[]`. A future version of the API may add a cursor.

Common URL query parameters:

- `match[]=<series_selector>`: Repeated series selector used to scope the
search. Optional.
- `search[]=<string>`: Repeated search string matched against names or values.
Multiple values use OR semantics. Optional.
- `fuzz_threshold=<number>`: Fuzzy threshold from 0 to 100. Optional. A value

@aknuds1 aknuds1 May 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest documenting the default:

Suggested change
- `fuzz_threshold=<number>`: Fuzzy threshold from 0 to 100. Optional. A value
- `fuzz_threshold=<number>`: Fuzzy threshold from 0 to 100. Optional. Default
is 0, the lowest fuzzy threshold.

of 0 is the lowest fuzzy threshold.
- `fuzz_alg=<subsequence | jarowinkler>`: Matching algorithm. Optional. Default
is `subsequence`.
- `case_sensitive=<bool>`: Toggle case-sensitive matching. Optional.

@aknuds1 aknuds1 May 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- `case_sensitive=<bool>`: Toggle case-sensitive matching. Optional.
- `case_sensitive=<bool>`: Toggle case-sensitive matching. Optional. Default
is `true`.

- `sort_by=<string>`: Sort mode. Supported values depend on the endpoint.

@aknuds1 aknuds1 May 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- `sort_by=<string>`: Sort mode. Supported values depend on the endpoint.
- `sort_by=<string>`: Sort mode. Supported values depend on the endpoint.
Optional; if unset, results are returned in natural order.

- `sort_dir=<asc | dsc>`: Sort direction. Optional. Only valid with
`sort_by=alpha`.
- `include_score=<bool>`: Include the relevance score in each result. Optional.

@aknuds1 aknuds1 May 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- `include_score=<bool>`: Include the relevance score in each result. Optional.
- `include_score=<bool>`: Include the relevance score in each result. Optional.
Default is `false`.

- `start=<rfc3339 | unix_timestamp>`: Start timestamp. Optional.
- `end=<rfc3339 | unix_timestamp>`: End timestamp. Optional.
- `limit=<number>`: Maximum number of returned results. Optional. Default is
100.
- `batch_size=<number>`: Preferred number of results per NDJSON batch.
Optional. Default is 100.

The `start` and `end` parameters narrow results to the selected time window.
Results may include values from series active slightly outside that window,
because Prometheus stores data in fixed-size blocks (typically 2 hours each).

Additional parameters for `/api/v1/search/metric_names`:

- `include_metadata=<bool>`: Include metric metadata in each result.

@aknuds1 aknuds1 May 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
- `include_metadata=<bool>`: Include metric metadata in each result.
- `include_metadata=<bool>`: Include metric metadata in each result. Optional.
Default is `false`.

- `sort_by=<alpha | score>`

Additional parameters for `/api/v1/search/label_names`:

- `sort_by=<alpha | score>`

Additional parameters for `/api/v1/search/label_values`:

- `label=<label_name>`: Label name whose values should be searched. Required.
- `sort_by=<alpha | score>`

This example searches metric names for autocomplete:

```bash
curl -g 'http://localhost:9090/api/v1/search/metric_names?search[]=http_req&sort_by=score&include_metadata=true&limit=5'
Comment thread
roidelapluie marked this conversation as resolved.
```

```json
{"results":[{"name":"http_requests_total","type":"counter","help":"Total HTTP requests."}]}
{"status":"success","has_more":false}
```

This example searches label values for the `instance` label within the `up`
metric:

```bash
curl -g 'http://localhost:9090/api/v1/search/label_values?label=instance&match[]=up&search[]=909&sort_by=score'
```

```json
{"results":[{"value":"localhost:9090"},{"value":"localhost:9091"}]}
{"status":"success","has_more":true}
```

## Querying exemplars

This is **experimental** and might change in the future.
Expand Down
Loading
Loading