# Prometheus v3.5.0

2025-07-14 · [Original release notes](https://github.com/prometheus/prometheus/releases/tag/v3.5.0) · [Full summary](https://ratatosk.io/en/releases/prometheus/v3.5.0)

Security 1 · Breaking changes 1 · Important fixes 16 · Enhancements 12 · Other changes 9

## Security (1)

### CORS responses now vary by Origin

- Affects: HTTP endpoints whose responses pass through `SetCORS` (Inferred)

This avoids cache reuse without varying on Origin.

> * [BUGFIX] API: Add HTTP `Vary: Origin` header to responses to avoid cache poisoning. #16008

## Breaking changes (1)

### String-label size encoding changed and length capped at 16MB

- Affects: setups using label names or values longer than 16MB (Inferred)

Previously, label sizes used varints; they now use one byte for sizes 0–254, or the value 255 followed by a three-byte little-endian size. Memory use is unchanged for sizes 0–127 and 2MB–16MB, one byte lower for 128–254 and 16384–2MB, and two bytes higher for 255–16383.

Evidence: “Instead of using varint to encode the size of each label, use a single byte for size 0-254, or a flag value of 255 followed by the size in 3 bytes little-endian.” https://github.com/prometheus/prometheus/pull/16069

> * [PERF] Labels: Simpler/faster stringlabels encoding. #16069

## Important fixes (16)

### Support for changing scalar parameters in PromQL range aggregations

- Affects: PromQL range queries using non-constant scalar parameters in aggregations such as `quantile` and `topk`

Results can now reflect the parameter value at each evaluation time across the query range.

> * [ENHANCEMENT] PromQL: support non-constant parameter in aggregations like `quantile` and `topk`. #16404

### Prometheus `GOGC` environment variable handling fixed

- Affects: Prometheus setups with `GOGC` set and no `runtime.gogc` field in the loaded config (Inferred)

Previously, when `GOGC` was set but the config had no `runtime` block, Prometheus ignored it and used the 75% default. It now uses the environment value whether or not a `runtime` block exists, unless `runtime.gogc` is set, which takes precedence.

> * [BUGFIX] Config: Respect GOGC environment variable if no "runtime" block exists. #16558

### PromQL `last_over_time` native histogram handling fixed

- Affects: queries of native histograms using `last_over_time`

For mixed samples with no float samples and histogram timestamps less than zero, it incorrectly returned a default float with `T=0, V=0.0`.

> * [BUGFIX] PromQL: Fix native histogram `last_over_time`. #16744

### PromQL histogram functions return no result on conflicting inputs

- Affects: setups using `histogram_fraction` or `histogram_quantile` on data with classic and native histograms for the same series at the same timestamp

In this conflict case, the functions could previously return a value where no result was expected. They now return no result and emit a mixed classic/native histogram warning.

> * [BUGFIX] PromQL: Don't emit a value from `histogram_fraction` or `histogram_quantile` if classic and native histograms are present at the same timestamp. #16552

### PromQL duration rounding fixed

- Affects: PromQL numeric durations in offsets, range selectors, and subquery ranges or steps

Durations such as `[1001ms]` could be rounded down to `[1s]`, causing selectors to miss samples on dense data. They are now rounded to the nearest nanosecond, so `[1001ms]` is represented as 1001 milliseconds.

> * [BUGFIX] PromQL: Incorrect rounding of `[1001ms]` to `[1s]` and similar. #16478

### PromQL counter-reset detection fixed for histogram queries

- Affects: native histogram queries using `rate`, `irate`, or `increase` with `histogram_count`, `histogram_sum`, or `histogram_avg`

When the last sample in the range triggered a counter reset from the previous sample, reset detection could fail, producing inconsistent or sometimes negative `histogram_count` and `histogram_sum` results.

> * [BUGFIX] PromQL: Fix inconsistent / sometimes negative `histogram_count` and `histogram_sum`. #16682

### PromQL `histogram_quantile` and `histogram_fraction` NaN handling fixed

- Affects: native histograms containing NaN observations used with `histogram_quantile` or `histogram_fraction`

With native histograms containing NaN observations, `histogram_quantile` could return 0 when all observations were NaN or the upper bucket bound when the quantile exceeded normal observations; `histogram_fraction` included NaNs in its calculation. Now the quantile returns NaN when it falls beyond all normal observations into NaNs, fractions exclude NaNs, and info-level annotations identify when NaNs affect the results.

> * [BUGFIX] PromQL: Improve handling of NaNs in native histograms. #16724

### PromQL unary-minus precedence in duration expressions fixed

- Affects: PromQL duration expressions enabled with `promql-duration-expr`

When a unary minus was followed by another operator, expressions such as `foo offset -1^1` could parse with the wrong precedence. The unary minus now binds to the duration first, so `foo offset -1^1` parses as `(foo offset -1) ^ 1` rather than `foo offset (-1^1)`.

> * [BUGFIX] PromQL: Fix unary operator precedence in duration expressions. #16713

### Prometheus and OpenMetrics text parser memory-corruption fix

- Affects: scrapes using the Prometheus or OpenMetrics text format (Inferred)

Parser label handling could retain a reference to parser-buffer data. The reproducer covers a data race when scraping overlaps with TSDB head compaction.

> * [BUGFIX] Scraping: Fix rare memory corruption bug. #16623

### NHCB parser resumes classic histogram conversion after exponential histograms

- Affects: protobuf scrapes containing native and NHCB-eligible classic histograms (Inferred)

Previously, after an exponential native histogram was encountered, subsequent eligible classic histograms were not converted to NHCB. Conversion now resumes when their labels differ from the exponential histogram; matching labels still inhibit conversion.

> * [BUGFIX] Scraping: continue handling custom-bucket histograms after an exponential histogram is encountered. #16720

### Service-discovery UI target counts use server totals

- Affects: service-discovery UI with `keep_dropped_targets` limiting retained dropped-target details

The UI indicates when further dropped-target details are not shown.

> * [ENHANCEMENT] UI: Better total target count display when using `keep_dropped_targets` option. #16604

### Prometheus rule-file validation before WAL loading

- Affects: Prometheus configurations with rule-file patterns

When configured patterns resolved to missing files or files contained invalid rules, parsing did not happen at this startup stage and could lead to systemd crash-restart loops. Now these cases produce a logged startup error and exit status 2 during the early check.

> * [ENHANCEMENT] Rules: Check that rules parse without error earlier at startup. #16601

### Prometheus fixes high memory use during TSDB WAL initialization

- Affects: Prometheus setups that initialize and load a TSDB WAL (Inferred)

During initialization and WAL loading, Go's default `GOGC` of 100% was in effect before Prometheus applied its configured value or its 75% default, causing higher memory usage. After the upgrade, the configured `GOGC` value or the 75% default is used during that phase.

> * [BUGFIX] Top-level: Update GOGC before loading TSDB. #16491

### PromQL aggregation positions corrected and invalid-expression panic fixed

- Affects: PromQL query parsing

The end position for an aggregation inside `ParenExpr` previously extended through the parenthesized expression, so position-based printing included an extra token; invalid expressions such as `sum(rate(` could panic on an out-of-range access. The end position now stops at the aggregate expression, and invalid expressions are handled without that panic.

> * [BUGFIX] PromQL: Fix reported parser position range in errors for aggregations wrapped in ParenExpr #16041 #16754

### Discovery lock-order deadlock fixed

- Affects: Prometheus service discovery (Inferred)

During shutdown, Prometheus could hang when `Manager.ApplyConfig()` and `Manager.cleaner()` acquired `Provider.mu` and `Manager.targetsMtx` in opposite orders. The routines now acquire those locks in a consistent order, addressing that deadlock scenario.

> * [BUGFIX] Discovery: Avoid deadlocks by taking locks in consistent order. #16587

### OTLP receiver defaults applied when `otlp:` is unset

- Affects: OTLP receiver configurations without an `otlp:` block

Previously, omitting the `otlp:` block left the OTLP defaults unapplied; the receiver now uses `DefaultOTLPConfig`, including `TranslationStrategy: UnderscoreEscapingWithSuffixes`.

> * [BUGFIX] OTLP: Default config not respected when `otlp:` block is unset. #16693

## Enhancements (12)

- Type and unit metadata labels with PromQL type annotations (To use: `--enable-feature=type-and-unit-labels`)
- PromQL timestamp-of-sample functions added (To use: The `experimental-promql-functions` feature flag must be enabled.)
- Global control for scraping classic histograms alongside native histograms (To use: Global `always_scrape_classic_histograms` option enabled)
- OTLP resource-attribute promotion options added (To use: Set `otlp.promote_all_resource_attributes: true` to promote all resource attributes. `ignore_resource_attributes` is only allowed with this setting, and it cannot be combined with `promote_resource_attributes`.)
- STACKIT Cloud service discovery
- Hetzner discovery server filtering with `label_selector` (To use: `role=hcloud`; `label_selector` is only available with that role.)
- Filtering rules by health, name, and labels on the `/rules` page
- Query statistics in the table query tab (To use: `stats: "true"` in table query requests)
- Optional fuzzy float comparison in `promtool test rules` (To use: `--fuzzy-compare` or `fuzzy_compare: true`)
- Experimental uncached I/O for chunk writing (To use: Enable `--enable-feature=use-uncached-io`; it is experimental and only available on Linux.)
- Graph view displays query warnings and info
- Azure AD remote-write supports the default managed identity (To use: Azure AD remote-write authentication with `managed_identity` configured and `client_id` blank)

## Other changes (9)

- Scrape-pool selector clears its selection during search
- `histogramStatsIterator` reuse reduces PromQL allocations
- PromQL reuses custom bucket storage for native histograms
- Reduced unnecessary re-renders on the `/targets` page
- Full-page re-render from the status menu fixed
- Kubernetes EndpointSlice discovery now uses a service `cache.Indexer`
- TSDB label-value fetching stops at the requested limit
- Concurrent scrape-pool reloads during configuration application
- PromQL averages switch to Kahan-summed means with range fallback

---

Summaries are produced by AI, so they can be wrong, and they are provided without warranty.
