RATATOSKRATATOSK
Sign in

Prometheus

v3.14.0Observability
Aug 18, 2026

ACTION 1CHECK 1PLAN 1OTHER 39

A maintenance release with an API deprecation, operator-visible removals and default changes, new capabilities, performance improvements, and correctness fixes across discovery, PromQL, TSDB, and other components. No security advisories or explicitly described security vulnerabilities are included.

Action needed (1)

  • breakingDefault support for PromQL duration expressions

    Duration expressions are enabled by default in PromQL. The promql-duration-expr feature flag is now a no-op.

Check if affected (1)

  • breakingThe __meta_hetzner_datacenter label, removed

    Applies if you use hcloud targets.

    The __meta_hetzner_datacenter label is removed from hcloud targets because it was removed from the Hetzner Cloud API.

Plan ahead (1)

  • deprecatedThe stats query parameter, deprecated for other valuesremoval date not announced

    Applies if you set stats to a value other than true or all.

    The stats query parameter is deprecated for values other than true and all on /api/v1/query and /api/v1/query_range. Other values still enable basic statistics but return a deprecation warning and will be rejected in the next major release.

All 39 other recorded changesfixes 22 · additions 8 · value changes 8 · constraints 1

fixes (22)

  • [CHANGE] API: /api/v1/status/config now correctly shows separator: "" and replacement: "" in relabel configs when explicitly set to empty, instead of omitting them.
  • [BUGFIX] Alerting: Fix 100% CPU usage on shutdown that could delay graceful shutdown and trigger timeout-based kills.
  • [BUGFIX] Discovery/AWS: Stop promtool check config from making AWS metadata service (IMDS) network calls when the region field is omitted in EC2, ECS, RDS, MSK, ElastiCache, and Lightsail service discovery configs.
  • [BUGFIX] Discovery/Docker: Set a request timeout for docker_sd and dockerswarm_sd on unix, npipe, and tcp hosts. Previously an unresponsive daemon could freeze discovery indefinitely, silently pinning targets to a stale snapshot.
  • [BUGFIX] Discovery/Docker: Fix panic in Docker Swarm service discovery when a service runs as a plugin or network-attachment.
  • [BUGFIX] Discovery/Docker: Fix discovery of IPv6-only containers.
  • [BUGFIX] PromQL: Fix case-insensitive regex label matchers silently dropping matching values.
  • [BUGFIX] PromQL: Fix mad_over_time returning 0 instead of NaN when the range contains a NaN sample.
  • [BUGFIX] Promtool: Accept --enable-feature=promql-binop-fill-modifiers in check rules, which previously rejected valid fill()/fill_left()/fill_right() expressions.
  • [BUGFIX] Remote write: Respect the AZURE_FEDERATED_TOKEN_FILE environment variable for workload identity authentication instead of hardcoding the token file path.
  • [BUGFIX] Rules: Clean up stale rule_group_last_rule_duration_sum_seconds and rule_group_last_restore_duration_seconds series when a rule group is removed or renamed on reload. Previously each reload leaked two series per dropped group, growing /metrics cardinality over time.
  • [BUGFIX] Scrape: Fix scrape manager spinning at 100% CPU on shutdown.
  • [BUGFIX] TSDB: Fix silent data loss and potential crash loop when stale_series_compaction_threshold is used in the config file.
  • [BUGFIX] TSDB: Fix potential data loss on restart when out-of-order ingestion is enabled and blocks are compacted.
  • [BUGFIX] TSDB: Fix prometheus_tsdb_head_stale_series over-counting and early eviction of series that change between float, integer histogram, and float histogram sample types.
  • [BUGFIX] TSDB: Fix goroutine and file handle leaks when Prometheus fails to open a corrupt TSDB. On Windows, the leaked directory handle also prevented TSDB directory removal.
  • [BUGFIX] TSDB: Fix out-of-order queries blocking compaction for hours, causing memory usage to grow.
  • [BUGFIX] TSDB: Fix deleted series causing missing samples and errors after restart.
  • [BUGFIX] TSDB: Fix native histogram data becoming incorrect after restart.
  • [BUGFIX] TSDB: Surface query errors that were previously silently discarded.
  • [BUGFIX] TSDB: Honour the configured float chunk encoding when compaction rewrites chunks; previously chunks encoded with --enable-feature=xor2-encoding could silently revert to XOR after compaction.
  • [BUGFIX] UI: Show the delete-series form on the TSDB Status page when --web.enable-admin-api is enabled.

additions (8)

  • [FEATURE] Discovery: Add Oracle Cloud Infrastructure compute service discovery (oci_sd_configs).
  • [FEATURE] PromQL: Add experimental start_timestamp(instant-vector) function returning the start timestamp of each sample in the given vector. Requires the use-start-timestamps feature flag.
  • [FEATURE] TSDB: Add experimental support for encoding start timestamps in histograms and float histograms. Hidden behind the histograms-st-encoding feature flag.
  • [ENHANCEMENT] OTLP: Emit a warning when OTLP attribute names collide into the same Prometheus label after sanitization (e.g. k8s.pod.name and k8s_pod_name both become k8s_pod_name), and expose the prometheus_api_otlp_translation_warnings_total counter labelled by category to track such warnings.
  • [ENHANCEMENT] Promtool: Add --remote-write.path flag to push metrics for backends that use a non-default remote-write endpoint.
  • [ENHANCEMENT] TSDB: Add prometheus_tsdb_head_native_histogram_series and prometheus_tsdb_head_native_histogram_buckets gauges tracking the number of native histogram series and buckets in the head.
  • [ENHANCEMENT] UI: Add syntax highlighting, autocompletion, and linting for PromQL duration expressions (step(), range(), min_of(), max_of()) in range selectors and subqueries.
  • [ENHANCEMENT] UI: Add copy button next to rule names on the Rules and Alerts pages.

value changes (8)

  • [CHANGE] PromQL: Promote first_over_time to stable. It no longer requires the promql-experimental-functions feature flag.
  • [ENHANCEMENT] Remote write: Forward histogram start timestamps in the remote write V2 protocol.
  • [ENHANCEMENT] UI: Improve rule group title contrast on the Rules page.
  • [PERF] Speed up regex label matchers matching a set of literal values (e.g. {job=~"foo|bar|baz"}).
  • [PERF] Remote read: Improve remote read throughput by removing unnecessary per-write flushing.
  • [PERF] Scrape: Parse text and OpenMetrics formats without recursion, preventing stack overflow from deeply nested or malicious exposition input.
  • [PERF] Scrape: Reduce native histogram scrape parsing allocations by ~49%.
  • [PERF] TSDB: Speed up queries on series with many in-memory chunks.

constraints (1)

  • [FEATURE] PromQL: Allow rate() and increase() to use start timestamps as an alternative for rate extrapolation. Hidden behind the use-start-timestamps feature flag.
Add Prometheus to your stack

A weekly email arrives when a release needs action. Like the breaking changes in this release.

Add to stack →
Official release note ↗All Prometheus releases →