Prometheus
v3.11.0ObservabilityPrometheus v3.11.0 ships significant new discovery capabilities, TSDB experimental features, and a critical retention bug fix that could cause data to be kept 1,000,000x longer than configured.
breakingFix TSDB retention unit bug before your disk fills up
A unit mismatch in the config file parser caused `storage.tsdb.retention.time` to be interpreted as 1,000,000x the intended value. If you set retention via the config file (not CLI flags), your TSDB may be holding far more data than expected. Upgrade to 3.11.0 immediately and verify disk usage post-upgrade. Also note: CLI flag values are now the fallback when retention is removed from config, so review your configuration carefully.
breakingMigrate Hetzner hcloud SD label references before July 2026
`__meta_hetzner_datacenter` and `__meta_hetzner_hcloud_datacenter_location*` labels are deprecated for hcloud roles and will stop working after July 1, 2026. Audit all relabeling configs and recording rules that reference these labels. Replace with `__meta_hetzner_hcloud_location` and `__meta_hetzner_hcloud_location_network_zone`. The robot role keeps the old label for backward compatibility.
enhancementCap TSDB disk usage with retention.percentage
The new `storage.tsdb.retention.percentage` setting lets you define a maximum percentage of available disk that TSDB can use. This is a cleaner safety net than estimating byte-based retention limits, especially in environments with variable data volumes. Consider pairing it with the existing time-based retention rather than replacing it — both constraints apply.
主な変更 (6)
- CRITICAL BUG: TSDB retention time unit mismatch caused retention to run 1e6x longer than configured — patch immediately
- Hetzner SD labels deprecated: `__meta_hetzner_datacenter` for hcloud role dies July 1, 2026; migrate relabeling configs now
- New AWS SD roles for Elasticache and RDS, plus Azure Workload Identity auth support
- New `storage.tsdb.retention.percentage` config to cap TSDB disk usage by percentage
- Experimental `fast-startup` flag writes series state to WAL dir, reducing restart time for large TSDB instances
- OTLP fix: ErrTooOldSample now returns HTTP 400 instead of 500, breaking infinite client retry loops