Prometheus
v3.9.0ObservabilityPrometheus 3.9.0 is a broad feature and behavior release covering native histogram collection, TSDB status reporting, APIs, metrics, PromQL, and the UI. The native histogram configuration change and the new TSDB status limit are the main concerns for deployments that use those areas.
Action needed (1)
breakingTSDB status endpoint statistics limit
The TSDB status endpoint now limits the response to a maximum of 10,000 sets of statistics.
Check if affected (1)
breakingThe
native-histogramfeature flag andscrape_native_histogramssettingApplies if you collect Native Histogram samples from exporters.
In version 3.9, Native Histograms are no longer experimental, and the
native-histogramfeature flag has no effect. Thescrape_native_histogramssetting must be enabled to collect Native Histogram samples from exporters.
All 34 other recorded changesfixes 16 · additions 9 · value changes 6 · constraints 3
fixes (16)
- - [BUGFIX] UI: PromQL autocomplete now shows the correct type and HELP text for OpenMetrics counters whose samples end in
_total. #17682 - - [BUGFIX] UI: Fixed codemirror-promql incorrectly showing label completion suggestions after the closing curly brace of a vector selector. #17602
- - [BUGFIX] UI: Query editor no longer suggests a duration unit if one is already present after a number. #17605
- - [BUGFIX] PromQL: Fix some "vector cannot contain metrics with the same labelset" errors when experimental delayed name removal is enabled. #17678
- - [BUGFIX] PromQL: Fix possible corruption of PromQL text if the query had an empty
ignoring()and non-empty grouping. #17643 - - [BUGFIX] PromQL: Fix resets/changes to return empty results for anchored selectors when all samples are outside the range. #17479
- - [BUGFIX] PromQL: Check more consistently for many-to-one matching in filter binary operators. #17668
- - [BUGFIX] PromQL: Fix collision in unary negation with non-overlapping series. #17708
- - [BUGFIX] PromQL: Fix collision in label_join and label_replace with non-overlapping series. #17703
- - [BUGFIX] PromQL: Fix bug with inconsistent results for queries with OR expression when experimental delayed name removal is enabled. #17161
- - [BUGFIX] PromQL: Ensure that
rate/increase/deltaof histograms results in a gauge histogram. #17608 - - [BUGFIX] PromQL: Do not panic while iterating over invalid histograms. #17559
- - [BUGFIX] TSDB: Reject chunk files whose encoded chunk length overflows int. #17533
- - [BUGFIX] TSDB: Do not panic during resolution reduction of invalid histograms. #17561
- - [BUGFIX] Remote-write Receive: Avoid duplicate labels when experimental type-and-unit-label feature is enabled. #17546
- - [BUGFIX] OTLP Receiver: Only write metadata to disk when experimental metadata-wal-records feature is enabled. #17472
additions (9)
- - [FEATURE] API: Add /api/v1/features for clients to understand which features are supported. #17427
- - [FEATURE] Promtool: Add
start_timestampfield for unit tests. #17636 - - [FEATURE] Promtool: Add
--format seriesjsonoption totsdb dumpto output just series labels in JSON format. #13409 - - [FEATURE] Add
--storage.flag for better interoperability with Thanos. #17435tsdb. delay-compact-file. path - - [FEATURE] UI: Add an option on the query drop-down menu to duplicate that query panel. #17714
- - [ENHANCEMENT]: TSDB: add flag
--storage.to configure TSDB Block Reload Interval. #16728tsdb. block-reload-interval - - [ENHANCEMENT] UI: Add graph option to start the chart's Y axis at zero. #17565
- - [ENHANCEMENT] Notifications: Add a histogram
prometheus_notifications_latency_histogram_secondsto complement the existing summary. #16637 - - [ENHANCEMENT] TSDB: New histogram
prometheus_tsdb_sample_ooo_delta, the distribution of out-of-order samples in seconds. Collected for all samples, accepted or not. #17477
value changes (6)
- In version 3.9, Native Histograms is no longer experimental, and the feature flag
native-histogramhas no effect. You must now turn on the config settingscrape_native_histogramsto collect Native Histogram samples from exporters. - - [ENHANCEMENT] SD: add a
configlabel with job name for mostprometheus_sd_refreshmetrics. #17138 - - [ENHANCEMENT] Remote-read: Validate histograms received via remote-read. #17561
- - [PERF] TSDB: Small optimizations to postings index. #17439
- - [PERF] Scraping: Speed up relabelling of series. #17530
- - [PERF] PromQL: Small optimisations in binary operators. #17524, #17519.
constraints (3)
- - [ENHANCEMENT] Scraping: Classic protobuf format no longer requires the unit in the metric name. #16834
- - [ENHANCEMENT] PromQL, Rules, SD, Scraping: Add native histograms to complement existing summaries. #17374
- - [ENHANCEMENT] Remote-write: Add custom scope support for AzureAD authentication. #17483
A weekly email arrives when a release needs action. Like the breaking changes in this release.