RATATOSKRATATOSK
로그인

Prometheus

v3.16.0Observability
2026년 11월 5일

ACTION 6CHECK 3OTHER 76

운영자가 확인할 기본값과 입력 검증이 바뀌었고, PromQL과 히스토그램 처리도 확장되었습니다. 저장소, API, 수집, 원격 쓰기 전반의 성능과 정확성 개선이 포함된 릴리스입니다.

조치 필요 (6)

  • breakingRemote Write 2.0 메타데이터 메트릭 노출 중단

    Remote Write 2.0 큐에서 더 이상 사용 중단된 prometheus_remote_storage_metadata_total, prometheus_remote_storage_metadata_failed_total, prometheus_remote_storage_metadata_retried_total, prometheus_remote_storage_metadata_bytes_total 메트릭을 노출하지 않습니다.

  • breaking중복 라벨 이름을 가진 exemplar 거부

    라벨 이름이 중복된 exemplar를 저장하지 않고 거부합니다. 이에 따라 /api/v1/query_exemplars에서 중복된 JSON 키가 반환되지 않습니다.

  • breakingSmoothed 및 Anchored 벡터 선택자 상시 활성화

    Smoothed 및 Anchored 벡터 선택자가 안정화되어 항상 활성화됩니다.

  • breakingtimeout 및 lookback_delta 오버플로 값 거부

    timeout과 lookback_delta가 int64 범위를 넘으면 음수 기간으로 변환하지 않고 거부합니다.

  • breaking미래 타임스탬프 exemplar 처리 수정

    현재 시각보다 10분 넘게 미래인 타임스탬프를 가진 exemplar를 저장하지 않습니다. 해당 exemplar 때문에 같은 시계열의 후속 exemplar가 차단되지도 않습니다.

  • breaking버킷 인덱스 오버플로 네이티브 히스토그램 거부

    버킷 인덱스에서 span 오프셋이 int32 범위를 넘는 네이티브 히스토그램을 저장하지 않고 거부합니다.

영향 확인 (3)

  • breaking실험적 검색 엔드포인트의 batch_size 상한 조정

    실험적 검색 엔드포인트를 사용하는 경우 적용됩니다.

    실험적 검색 엔드포인트에서 batch_size가 1000을 넘지 않도록 제한됩니다.

  • breakingTLS 활성화 시 --web.external-url 기본값 변경

    --web.config.file을 사용하는 경우 적용됩니다.

    --web.config.file로 TLS를 활성화하면 --web.external-url의 기본값이 https로 설정됩니다.

  • breakingNHCB 변환 시 합계 전용 클래식 히스토그램 거부

    NHCB 변환을 사용하는 경우 적용됩니다.

    NHCB 변환 중 합계만 있는 클래식 히스토그램을 거부합니다.

그 외 기록된 변경 76건 전체fixes 52 · value changes 14 · additions 10

fixes (52)

  • Abort failed streaming responses without writing duplicate headers or appending plain-text errors to the protobuf stream
  • Fix loss of out-of-order samples when a WAL checkpoint drops a series record that the out-of-order write-behind log still needs
  • Fix queries that seek within a head chunk returning samples committed after the query started
  • Preserve mapped chunks when WAL replay merges multiple references for the same series, preventing missing out-of-order samples and duplicate in-order chunks after restart
  • Fix auto-reload logic to reload after reverting a failed configuration change
  • Do not match metric family metadata for doubled _total and _info suffixes in experimental search API.
  • Resolve metric family metadata for suffixed metric names in search results, in experimental search API.
  • Describe each ElastiCache cluster once instead of twice for clusters that are not replication group members.
  • Allow installing promtool with version-suffixed go install commands
  • Fix a panic in Azure service discovery when a VM tag has a null value.
  • fix corruption of float native histograms.
  • Fix ts_of_last_over_time for samples before the Unix epoch.
  • Fix ConvertNHCBToClassic inflating classic bucket counts for integer native histograms whose spans have gaps.
  • Preserve rule and TSDB block values in JSON logs.
  • Count only out-of-order exemplars in prometheus_api_otlp_out_of_order_exemplars_total, not every rejected exemplar.
  • Do not drop exemplars that have a trace_id or span_id filtered attribute in addition to a trace or span ID.
  • Do not emit le="+Inf" twice when a histogram's explicit bounds end at +Inf, and warn about the bound.
  • Stop dropping exemplars that arrive out of timestamp order within a data point.
  • Attach histogram exemplars to matching classic _bucket series and preserve them when scraping both native and classic histograms.
  • Avoid intermediate overflow in rad() and deg() conversions.
  • Consistently ignore and drop __type__ and __unit__ labels in without aggregations, absent(), info(), and binary operator fill matching.
  • Do not panic when printing the partial AST returned alongside a parse error.
  • Drop __type__ and __unit__ in binary operations so they are transparent to users, similar to how __name__ is handled.
  • Fix ==, !=, changes() and count_values on native histograms giving wrong results inside histogram_count(), histogram_sum() and histogram_avg().
  • Fix histogram_fraction returning wrong results for native histograms with custom buckets when a bucket boundary is exactly 0.
  • Fix histogram_quantile and histogram_quantiles interpolating within the wrong bucket for native histograms whose sum is NaN.
  • Fix lexing of subqueries whose range is a duration keyword expression, e.g. foo[range():1s].
  • Fix warning and error locations for duration expressions min_of and max_of.
  • Ignore NaN classic histogram bucket boundaries in histogram_quantile, histogram_quantiles and histogram_fraction and return correct value instead of NaN if there are normal buckets.
  • Preserve infinite values at exact quantile ranks.
  • Preserve original operand sides for experimental fill modifiers with group_right matching.
  • Preserve subquery step expressions when printing queries after preprocessing.
  • Range vector functions no longer raise a spurious "vector cannot contain metrics with the same labelset" error when series have different metric names but the same remaining labels, as long as their timestamps do not overlap within the evaluated range window
  • do not count subquery materialized samples in total samples statistics
  • Prevent future-dated exemplars from scaling shards down to the minimum.
  • Prevent future sent timestamps from advancing WAL truncation beyond the minimum retention window.
  • Restore firing alerts with for durations shorter than --rules.alert.for-grace-period after a restart.
  • fix failing scrape when float histogram is emitted over protobuf with count, zero count and buckets all zero, for example a result of recording rate(x[1m]) over constant histogram.
  • Return the lexically smallest label values when limit is set on /api/v1/search/label_values, instead of an arbitrary subset
  • Fix WAL watcher permanently skipping checkpoint series garbage collection when lagging behind head truncation.
  • Avoid panics when the requested time range has no participating storage.
  • Fix a race in selected-series and stale-series compaction where a sample committed after the block was written but before the series was evicted from the head could be silently lost.
  • Fix a rare case where a sample could be lost during selected-series or stale-series compaction if an unrelated write to the same series ran cleanup that erased the evidence needed to protect it.
  • Fix spurious "unknown series references" warnings during WAL replay after a restart, when the only blocks on disk were produced by selected-series or stale-series compaction.
  • Preserve series identities needed to recover exemplars after compaction and restart.
  • fix no passing metadata-wal-records to TSDB from main feature flag.
  • Preserve IPv6 addresses with dotted zone identifiers in stripDomain.
  • Don't automatically open an autocomplete dropdown after closing a function call body or aggregation operator with a closing parenthesis.
  • Fix HTTP/2 negotiation silently falling back to HTTP/1.1 under Go 1.27+ when TLS is enabled with http2: true. The server now correctly advertises h2 via ALPN and multiplexes concurrent requests over a single connection.
  • Fix double-counting of failed samples in the PrometheusRemoteStorageFailures alert denominator.
  • Honor --enable-feature when evaluating rules in test rules.
  • Ensure UTF-8 works with the experimental OpenMetrics 2.0

value changes (14)

  • Subquery samples are now included in the query TotalSamples and TotalSamplesPerStep statistics, and SamplesRead now counts reads from subqueries with an @ modifier in range queries.
  • Count Remote Write 2.x samples and histograms ingested without metadata in prometheus_api_remote_write_without_metadata_appended_samples_total even when metadata-wal-records is disabled.
  • Warn when participating remote-read storage does not support search.
  • Cache repeated OTLP histogram metric names.
  • Reuse classic histogram and summary expansion buffers across scrapes.
  • Decode only histogram sum/count (not buckets) for timestamp(), start_timestamp() and absent().
  • Only select the fixed window for subqueries with an @ modifier.
  • Deduplicate per-series metadata structs in memory with unique.Handle.
  • Avoid some copies of scrape response bodies.
  • Improve XOR2 compression and append performance by selecting value windows based on encoding cost.
  • Reduce memory used writing and reading checkpoint metadata records.
  • Convert classic histograms to NHCB directly in the OpenMetrics 2.0 parser without wrapping in NHCBParser.
  • Reduce OpenMetrics 2.0 parser allocations and CPU overhead by lazily constructing composite series and exemplar labels.
  • Increment prometheus_tsdb_wal_corruptions_total when WAL corruption is detected during checkpoint creation, so the metric reflects corruption found at compaction time rather than only after a restart-triggered repair.

additions (10)

  • Allow overriding PromQL Start Timestamp processing on a per-query basis using the X-Prometheus-Use-Start-Timestamps HTTP header.
  • Support metadata-wal-records in Prometheus Agent mode.
  • Add experimental integral() function for computing integral-over-time of range vectors behind the --enable-feature=promql-experimental-functions flag.
  • Add experimental query cost estimation, actual cost reporting, and configurable execution limits, enabled with --enable-feature=query-cost.
  • Use start timestamps to detect counter resets when float timeseries are queried using extended range selector modifiers (anchored, smoothed). This feature is hidden behind use-start-timestamps feature flag.
  • print test filename during promtool tests
  • Allow unquoted Unicode letters and dots behind a feature flag
  • Add native histogram prometheus_remote_read_handler_gate_wait_duration_seconds to measure concurrency-gate wait duration, including requests canceled while waiting.
  • Add experimental flag convert_nhcb_to_classic to convert NHCB to classic histograms for external systems that dont support NHCBs.
  • Add --allow-incompatible-block-duration so users can specify any max-block-duration instead of being restricted to the default Prometheus block durations.
Prometheus 스택에 추가

조치가 필요한 릴리스가 나왔을 때 주간 메일로 알려드립니다. 이번 릴리스의 브레이킹 체인지 같은 것들입니다.

스택에 추가 →
공식 릴리스 노트 ↗Prometheus 릴리스 전체 →