RATATOSKRATATOSK
Sign in

Prometheus

v3.12.0Observability
May 29, 2026

Prometheus 3.12.0 ships two security patches (Remote Write DoS, STACKIT SD secret leak), fixes a WAL race in Agent mode, and cuts TSDB range query CPU with a quadratic-to-constant head chunk lookup fix.

  • securityPatch two security fixes now — especially if using STACKIT SD

    Two CVEs patched in this release. First: Remote Write now rejects snappy-compressed payloads where the declared decoded size exceeds 32 MB — this closes a DoS vector against your remote-write receiver endpoint. Second: STACKIT SD was leaking secrets in plaintext via the `/-/config` endpoint (GHSA-39j6-789q-qxvh). If you use STACKIT SD, rotate any credentials that may have been exposed before upgrading.

  • breakingAgent mode WAL race and remote_write panic bug fixed — upgrade Agent deployments

    A race condition in the agent appender could produce duplicate in-memory series and duplicate WAL records when concurrent appends target the same label set. If you run Prometheus in Agent mode under high write concurrency, this bug could silently corrupt your WAL. Upgrade to 3.12.0 to fix it. Also, `remote_write` queue_config fields are now validated at load time — misconfigurations that previously caused silent runtime panics will now fail at startup, which is the right behavior but means you should test configs before rolling out.

  • enhancementTSDB range query CPU cut and auto-reload-config is now stable

    TSDB head chunk lookup in range queries drops from quadratic to constant time, and mmap operations now skip series that don't need work. At production scale with large head chunks, this can meaningfully cut CPU. No config changes needed — just upgrade. Separately, `auto-reload-config` is now stable (no longer experimental), so you can drop any caveats around it in runbooks.

Key changes (18)

  • Security: Remote Write rejects snappy payloads with decoded size over 32 MB (DoS fix); STACKIT SD secret leak via /-/config endpoint patched (GHSA-39j6-789q-qxvh)
  • TSDB performance: head chunk range query lookup is now O(1) instead of O(n²); mmap skips clean series, reducing CPU at scale
  • PromQL: new experimental functions start(), end(), range(), step(); rate()/irate()/increase()/resets() updated to use start timestamps behind the use-start-timestamps feature flag
  • Service Discovery: DigitalOcean Managed Databases and Outscale VM added; AWS EC2 SD gains IPv6 support; AWS SD gets optional external_id for ECS/MSK/RDS/ElastiCache
  • Bug fixes: agent WAL race condition patched; scrape panics on malformed histograms fixed; TSDB native histogram query panic fixed; remote_write queue_config now validated at startup
  • UI: time series deletion and tombstone cleanup now available from the Status menu
  • auto-reload-config promoted to stable
  • OTLP gzip body size now capped to prevent decompression abuse
  • SD target updates propagate faster via dynamic backoff instead of static 5s interval
  • Consul SD health_filter fix for Catalog-only fields like ServiceTags
  • prometheus_sd_refresh and prometheus_sd_discovered_targets metrics cleaned up when scrape jobs are removed
  • PromQL warns when sort/sort_by_label used in range queries (no-op in that context)
  • sort/sort_by_label warning in range queries, NaN/infinite duration expressions now rejected
  • Scrape: st-synthesis feature flag added to synthesize start timestamps for cumulative metrics when using Remote Write 2.0
  • promtool query instant gains --header flag
  • aix/ppc64 compilation target added
  • /api/v1/status/self_metrics endpoint added
  • Tracing: OTLP HTTP insecure startup failure fixed
Source ↗All Prometheus releases →