RATATOSK
← All releases

Prometheusv3.5.0

Security 1Breaking changes 1Important fixes 16Enhancements 12Other changes 9

Get Prometheus alerts

Email is sent weekly only when there are changes to check; critical security advisories are sent immediately.

Security1

CORS responses now vary by Origin

AffectsHTTP endpoints whose responses pass through SetCORS(Inferred)

This avoids cache reuse without varying on Origin.

Show release-note lines
* [BUGFIX] API: Add HTTP `Vary: Origin` header to responses to avoid cache poisoning. #16008
github.com/prometheus/prometheus/releases/tag/v3.5.0

Breaking changes1

String-label size encoding changed and length capped at 16MB

Affectssetups using label names or values longer than 16MB(Inferred)

Previously, label sizes used varints; they now use one byte for sizes 0–254, or the value 255 followed by a three-byte little-endian size. Memory use is unchanged for sizes 0–127 and 2MB–16MB, one byte lower for 128–254 and 16384–2MB, and two bytes higher for 255–16383.

Evidence“Instead of using varint to encode the size of each label, use a single byte for size 0-254, or a flag value of 255 followed by the size in 3 bytes little-endian.” prometheus/prometheus/pull/16069 ↗
Show release-note lines
* [PERF] Labels: Simpler/faster stringlabels encoding. #16069
github.com/prometheus/prometheus/releases/tag/v3.5.0

Important fixes16

  • Support for changing scalar parameters in PromQL range aggregations
    AffectsPromQL range queries using non-constant scalar parameters in aggregations such as quantile and topk

    Results can now reflect the parameter value at each evaluation time across the query range.

    * [ENHANCEMENT] PromQL: support non-constant parameter in aggregations like `quantile` and `topk`. #16404
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • Prometheus GOGC environment variable handling fixed
    AffectsPrometheus setups with GOGC set and no runtime.gogc field in the loaded config(Inferred)

    Previously, when GOGC was set but the config had no runtime block, Prometheus ignored it and used the 75% default. It now uses the environment value whether or not a runtime block exists, unless runtime.gogc is set, which takes precedence.

    * [BUGFIX] Config: Respect GOGC environment variable if no "runtime" block exists. #16558
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • PromQL last_over_time native histogram handling fixed
    Affectsqueries of native histograms using last_over_time

    For mixed samples with no float samples and histogram timestamps less than zero, it incorrectly returned a default float with T=0, V=0.0.

    * [BUGFIX] PromQL: Fix native histogram `last_over_time`. #16744
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • PromQL histogram functions return no result on conflicting inputs
    Affectssetups using histogram_fraction or histogram_quantile on data with classic and native histograms for the same series at the same timestamp

    In this conflict case, the functions could previously return a value where no result was expected. They now return no result and emit a mixed classic/native histogram warning.

    * [BUGFIX] PromQL: Don't emit a value from `histogram_fraction` or `histogram_quantile` if classic and native histograms are present at the same timestamp. #16552
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • PromQL duration rounding fixed
    AffectsPromQL numeric durations in offsets, range selectors, and subquery ranges or steps

    Durations such as [1001ms] could be rounded down to [1s], causing selectors to miss samples on dense data. They are now rounded to the nearest nanosecond, so [1001ms] is represented as 1001 milliseconds.

    * [BUGFIX] PromQL: Incorrect rounding of `[1001ms]` to `[1s]` and similar. #16478
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • PromQL counter-reset detection fixed for histogram queries
    Affectsnative histogram queries using rate, irate, or increase with histogram_count, histogram_sum, or histogram_avg

    When the last sample in the range triggered a counter reset from the previous sample, reset detection could fail, producing inconsistent or sometimes negative histogram_count and histogram_sum results.

    * [BUGFIX] PromQL: Fix inconsistent / sometimes negative `histogram_count` and `histogram_sum`. #16682
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • PromQL histogram_quantile and histogram_fraction NaN handling fixed
    Affectsnative histograms containing NaN observations used with histogram_quantile or histogram_fraction

    With native histograms containing NaN observations, histogram_quantile could return 0 when all observations were NaN or the upper bucket bound when the quantile exceeded normal observations; histogram_fraction included NaNs in its calculation. Now the quantile returns NaN when it falls beyond all normal observations into NaNs, fractions exclude NaNs, and info-level annotations identify when NaNs affect the results.

    * [BUGFIX] PromQL: Improve handling of NaNs in native histograms. #16724
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • PromQL unary-minus precedence in duration expressions fixed
    AffectsPromQL duration expressions enabled with promql-duration-expr

    When a unary minus was followed by another operator, expressions such as foo offset -1^1 could parse with the wrong precedence. The unary minus now binds to the duration first, so foo offset -1^1 parses as (foo offset -1) ^ 1 rather than foo offset (-1^1).

    * [BUGFIX] PromQL: Fix unary operator precedence in duration expressions. #16713
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • Prometheus and OpenMetrics text parser memory-corruption fix
    Affectsscrapes using the Prometheus or OpenMetrics text format(Inferred)

    Parser label handling could retain a reference to parser-buffer data. The reproducer covers a data race when scraping overlaps with TSDB head compaction.

    * [BUGFIX] Scraping: Fix rare memory corruption bug. #16623
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • NHCB parser resumes classic histogram conversion after exponential histograms
    Affectsprotobuf scrapes containing native and NHCB-eligible classic histograms(Inferred)

    Previously, after an exponential native histogram was encountered, subsequent eligible classic histograms were not converted to NHCB. Conversion now resumes when their labels differ from the exponential histogram; matching labels still inhibit conversion.

    * [BUGFIX] Scraping: continue handling custom-bucket histograms after an exponential histogram is encountered. #16720
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • Service-discovery UI target counts use server totals
    Affectsservice-discovery UI with keep_dropped_targets limiting retained dropped-target details

    The UI indicates when further dropped-target details are not shown.

    * [ENHANCEMENT] UI: Better total target count display when using `keep_dropped_targets` option. #16604
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • Prometheus rule-file validation before WAL loading
    AffectsPrometheus configurations with rule-file patterns

    When configured patterns resolved to missing files or files contained invalid rules, parsing did not happen at this startup stage and could lead to systemd crash-restart loops. Now these cases produce a logged startup error and exit status 2 during the early check.

    * [ENHANCEMENT] Rules: Check that rules parse without error earlier at startup. #16601
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • Prometheus fixes high memory use during TSDB WAL initialization
    AffectsPrometheus setups that initialize and load a TSDB WAL(Inferred)

    During initialization and WAL loading, Go's default GOGC of 100% was in effect before Prometheus applied its configured value or its 75% default, causing higher memory usage. After the upgrade, the configured GOGC value or the 75% default is used during that phase.

    * [BUGFIX] Top-level: Update GOGC before loading TSDB. #16491
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • PromQL aggregation positions corrected and invalid-expression panic fixed
    AffectsPromQL query parsing

    The end position for an aggregation inside ParenExpr previously extended through the parenthesized expression, so position-based printing included an extra token; invalid expressions such as sum(rate( could panic on an out-of-range access. The end position now stops at the aggregate expression, and invalid expressions are handled without that panic.

    * [BUGFIX] PromQL: Fix reported parser position range in errors for aggregations wrapped in ParenExpr #16041 #16754
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • Discovery lock-order deadlock fixed
    AffectsPrometheus service discovery(Inferred)

    During shutdown, Prometheus could hang when Manager.ApplyConfig() and Manager.cleaner() acquired Provider.mu and Manager.targetsMtx in opposite orders. The routines now acquire those locks in a consistent order, addressing that deadlock scenario.

    * [BUGFIX] Discovery: Avoid deadlocks by taking locks in consistent order. #16587
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • OTLP receiver defaults applied when otlp: is unset
    AffectsOTLP receiver configurations without an otlp: block

    Previously, omitting the otlp: block left the OTLP defaults unapplied; the receiver now uses DefaultOTLPConfig, including TranslationStrategy: UnderscoreEscapingWithSuffixes.

    * [BUGFIX] OTLP: Default config not respected when `otlp:` block is unset. #16693
    github.com/prometheus/prometheus/releases/tag/v3.5.0

Enhancements12

  • Type and unit metadata labels with PromQL type annotations

    To use --enable-feature=type-and-unit-labels

    * [FEATURE] PromQL: Add experimental type and unit metadata labels, behind feature flag `type-and-unit-labels`. #16228 #16632 #16718 #16743
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • PromQL timestamp-of-sample functions added

    To use The experimental-promql-functions feature flag must be enabled.

    * [FEATURE] PromQL: Add `ts_of_(min|max|last)_over_time`, behind feature flag `experimental-promql-functions`. #16722 #16733
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • Global control for scraping classic histograms alongside native histograms

    To use Global always_scrape_classic_histograms option enabled

    * [FEATURE] Scraping: Add global option `always_scrape_classic_histograms` to scrape a classic histogram even if it is also exposed as native. #16452
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • OTLP resource-attribute promotion options added

    To use Set otlp.promote_all_resource_attributes: true to promote all resource attributes. ignore_resource_attributes is only allowed with this setting, and it cannot be combined with promote_resource_attributes.

    * [FEATURE] OTLP: New config options `promote_all_resource_attributes` and `ignore_resource_attributes`. #16426
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • STACKIT Cloud service discovery
    * [FEATURE] Discovery: New service discovery for STACKIT Cloud. #16401
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • Hetzner discovery server filtering with label_selector

    To use role=hcloud; label_selector is only available with that role.

    * [ENHANCEMENT] Hetzner SD: Add `label_selector` to filter servers. #16512
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • Filtering rules by health, name, and labels on the /rules page
    * [ENHANCEMENT] UI: Add simple filtering on the `/rules` page. #16605
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • Query statistics in the table query tab

    To use stats: "true" in table query requests

    * [ENHANCEMENT] UI: Display query stats in hover tooltip over table query tab. #16723
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • Optional fuzzy float comparison in promtool test rules

    To use --fuzzy-compare or fuzzy_compare: true

    * [ENHANCEMENT] Promtool: Optional fuzzy float64 comparison in rules unittests. #16395
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • Experimental uncached I/O for chunk writing

    To use Enable --enable-feature=use-uncached-io; it is experimental and only available on Linux.

    * [PERF] TSDB: Optionally use Direct IO for chunks writing. #15365
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • Graph view displays query warnings and info
    * [BUGFIX] UI: Add query warnings and info to graph view. #16753 #16759
    github.com/prometheus/prometheus/releases/tag/v3.5.0
  • Azure AD remote-write supports the default managed identity

    To use Azure AD remote-write authentication with managed_identity configured and client_id blank

    * [BUGFIX] Remote-write: For Azure AD auth, allow empty `client_id` to suppport system assigned managed identity. #16421
    github.com/prometheus/prometheus/releases/tag/v3.5.0
Other changes9

Summaries are produced by AI, so they can be wrong, and they are provided without warranty.