RATATOSK
← All releases

NATSv2.11.11

Important fixes 28Enhancements 7Dependency updates 6Other changes 16

Get NATS alerts

Email is sent weekly only when there are changes to check; critical security advisories are sent immediately.

Check before upgrading

Upgrade guide1
2.11 upgrade guide for 2.10.x compatibility
1 items from linked documents
  • Guides for upgrading between minor versions
Original release notes
Refer to the [2.11 Upgrade Guide](https://docs.nats.io/release-notes/whats_new/whats_new_211) for backwards compatibility notes with 2.10.x.
github.com/nats-io/nats-server/releases/tag/v2.11.11

Important fixes28Silent failure2

  • JetStream stream catch-up health reporting fixedSilent failure
    AffectsJetStream streams that are catching up

    While a stream was catching up, its health check reported group node unhealthy because the Raft state did not advance until catch-up finished. The health check now checks for catch-up first and reports stream catching up.

    - Healthchecks now correctly report when streams are catching up, instead of showing them as unhealthy (#7535)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • JetStream filestore tombstone loss during secure erase fixed
    AffectsJetStream filestore setups using secure erase

    When a message before tombstones in a block was erased, an ungraceful shutdown requiring an index rebuild could lose the tombstones and bring deleted messages back. The filestore now preserves those tombstones in this case.

    - The filestore no longer loses tombstones when using secure erase (#7384)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • JetStream filestore recovery sequence rollback fixed
    AffectsJetStream filestore setups recovering blocks containing only tombstones

    After messages were acknowledged, their tombstones overflowed into new blocks, and the original message block was removed, an ungraceful restart could use the minimum known tombstone sequence and timestamp, rolling stream sequences back. With tombstones across multiple blocks, recovery could also assign an empty stream a first sequence that was too low; recovery now uses the maximum known tombstone sequence and timestamp in this case.

    - The filestore no longer loses the last sequence when recovering blocks containing only tombstones (#7384)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • Stream replica handling of out-of-order SkipMsg fixed
    Affectsstreams with replicas

    When a stream replica received an out-of-order SkipMsg, it could mark and delete a message that should not have been skipped, desynchronizing the stream. Skip sequences are now checked for ordering before they are applied, preventing this desynchronization during catchups.

    - Skipped message sequences are now checked for ordering before apply, fixing a potential stream desync on catchups (#7400)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • Filestore erased-message gap tracking fixed
    AffectsJetStream streams using the filestore(Inferred)

    When a gap in filestore messages ended with an erased final message, the filestore failed to register the erased slots as deletes, causing constant cache invalidation, slow reads or catchup, and incorrect delete tracking. During state rebuilding, it now registers the missing sequence numbers as deletes.

    - Skipped message sequences now correctly detect gaps from erased message slots, fixing potential cache issues, slow reads and issues with catchups (#7399, #7401)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • Filestore sequence-gap deletion tracking fixed
    AffectsJetStream streams using the filestore

    When a block with multiple SkipMsgs was empty at the start or fully empty, rebuilding could record sequence gaps in the dmap as deleted messages, even though the skips only advanced sequence numbers. Rebuilding now leaves those gaps out of the deleted-message map.

    - Skipped message sequences now correctly detect gaps from erased message slots, fixing potential cache issues, slow reads and issues with catchups (#7399, #7401)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • Raft peer activity reporting corrected
    AffectsRaft groups and peer-activity reporting in APIs, asset information, and monitoring endpoints

    After leader changes or cluster resizes, old active timestamps could remain unchanged, making “last seen” and “active” values misleading. This release tracks peer activity more consistently and clears stale timestamps on leader changes, so reported values can differ from the previous release.

    - Raft groups now report peer activity more consistently, fixing some cases where asset info and monitoring endpoints may report misleading values after leader changes (#7402)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • Raft follower WAL truncation after catchup fixed
    AffectsRaft followers running parallel catchups

    With multiple catchups running in parallel, a follower could accept a late append from another catchup after completion and truncate WAL entries already reported as eligible to commit. After catchup completes, unexpected catchup entries no longer truncate the WAL.

    - Raft groups will no longer permit truncations from unexpected catchup entries if the catchup is completed (#7424)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • NATS header suffix-matching bug fixed
    Affectsmessages with header keys that share a common suffix

    When getting, removing, or setting Nats-Msg-Id after Previous-Nats-Msg-Id was present, the operation could use the earlier key and value instead.

    - A couple of issues related to header handling have been fixed (#7465)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • setHeader now updates existing headers correctly
    Affectscode using setHeader to update existing headers

    Previously, calling setHeader for k on an existing k:v header whose value was not prefixed with whitespace added another header instead of updating the existing one.

    - A couple of issues related to header handling have been fixed (#7465)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • JetStream empty-stream NoWait response fixed
    AffectsJetStream consumers using NoWait requests without expiry

    When such a request reached the end of an empty stream without delivering messages, it did not receive 404 No Messages; it now does.

    - No-wait requests now return a 400 No Messages response correctly if the stream is empty (#7466)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • Filestore tail hole-filling sequence error fixed
    AffectsJetStream filestore

    When filling holes at the end of a block, an off-by-one error could mark one sequence as deleted and miss the last sequence.

    - An off-by-one when detecting holes at the end of a filestore block has been fixed (#7508)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • False Raft quorum-loss reports fixed during leader shutdown and stepdownSilent failure
    AffectsRaft groups led by the server

    Previously, shutting down a leader with LDM or having it step down cleared peer timestamps, making its Raft groups report lost quorum, print “NO quorum, stalled,” and send an advisory. The leader now retains its peer timestamps after stepping down; timestamps for other followers can still be cleared once the new leader comes online.

    - Raft groups should no longer report no quorum incorrectly when shutting down (#7522)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • JetStream filestore end-of-block hole detection fixed
    AffectsJetStream filestore(Inferred)

    When detecting holes at a block's end, a tombstone could be treated as the last sequence, causing deletes to be missed. The filestore now bases sequence handling on the last known-good sequence.

    - An off-by-one when detecting holes at the end of a filestore block has been fixed (#7525)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • NATS server prefers binary snapshots on new route connections
    Affectsnodes on new route connections to servers running v2.10.0 or later

    Before STATSZ arrived on a newly established route, the server could fall back to the legacy format, whose []uint64 representation of interior deletes could use disastrous amounts of memory for streams with many interior deletes.

    - Binary stream snapshots are now preferred by default for nodes on new route connections (#7479)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • Interest-based stream replicas recover from desynchronization
    Affectsclustered JetStream streams using InterestPolicy

    When consumer changes establish interest at different logical times across servers, replicas could desynchronize and store different initial messages. Servers can now detect this noInterest desynchronization and remove leading messages with no interest, though the underlying ordering race can still occur.

    - Improve interest detection when consumers are created or deleted across different servers (#7440)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • Per-gateway hop header corruption fixed
    Affectssetups using message tracing across gateways

    With message tracing enabled, setting the hop header for each gateway could corrupt headers. The header is now set per gateway without the described corruption.

    - When using message tracing, header corruption when setting the hop header has been fixed (#7443)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • Lame-duck connection-limit errors fixed
    Affectsaccounts with a maximum-connections limit on servers using lame-duck mode

    When account connections were near their limit during lame-duck mode, closed connections remained in the count and reconnecting clients were counted twice, so clients not yet reconnected could receive a maximum account active connections exceeded error. Connection accounting now avoids that double-counting; during shutdown with s.Shutdown(), remaining clients can still be closed without accounting.

    - Shutting down a server using lame-duck mode should no longer result in max connection exceeded errors (#7527)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • JetStream API handler races and nil-meta panics fixed
    AffectsJetStream API handlers, including consumer-info handling

    Some handlers accessed cc.meta without locking it, creating race conditions, and could panic when it was nil. The affected handlers no longer access it unlocked, and the consumer-info handler checks for nil before use.

    - Race conditions and potential panics fixed in the handling of some JetStream API handlers (#7380)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • Filestore TTL and scheduling state writes honor sync_always
    AffectsJetStream filestores using sync_always for TTL or scheduling state files
    - The filestore now correctly obeys `sync_always` for writing TTL and scheduling state files (#7385)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • Filestore locks released on message-erasure errors
    AffectsJetStream streams using the filestore(Inferred)

    If slotInfo or mb.eraseMsg returned an error during message erasure, the filestore could return without releasing the relevant locks; it now releases them before returning the error.

    - The filestore will now correctly release locks when erasing messages returns an error (#7431)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • JetStream filestore shutdown race fixed
    AffectsJetStream filestore

    During shutdown, if the lock was re-acquired while index.db was being generated, fs.StoreMsg could proceed after the index was written, leaving an invalid index on restart.

    - Fixed a race condition in the filestore that could happen between storing messages and shutting down (#7496)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • Parallel stream restoration panic fixed
    Affectsparallel stream restoration during startup

    During startup, ats.Register() could run before the access-time service had registered a value, causing an access time service not running panic. AccessTime() now stores and returns an initial timestamp in this case, which may be slightly stale on startup.

    - A panic that could occur when recovering streams in parallel has been fixed (#7503)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • JetStream monitors use the correct Raft group after stream recreation
    AffectsJetStream metalayer recovery with recreated streams and associated consumers(Inferred)

    When a stream and consumer were in a meta snapshot and the stream was deleted and recreated with a different config, their monitor goroutines could use a Raft node from the deleted group and fail to function properly.

    - Fixed a bug on metalayer recovery where stream and consumer monitor goroutines for recreated assets would run with the wrong Raft group (#7510)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • Snapshot installation fixed during R1-to-R3 stream scale-up
    AffectsJetStream streams scaled up from R1 to R3

    If JetStream catchup failed partway during a scale-up, incomplete peers could try to become leader and desynchronize the stream. The snapshot is now installed so recovery can process it, and new peers cannot become leader until they are fully synced.

    - Scaling up an asset from R1 now results in an installed snapshot, allowing recovery after restart if interrupted, avoiding a potential desync (#7509)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • JetStream recovery health checks no longer include deleted consumers
    AffectsJetStream metalayer snapshot recovery

    When a consumer in the meta snapshot was deleted during recovery, it could reappear in the health check and cause a consumer not found error.

    - Consumers that existed in a metalayer snapshot but were deleted on recovery will no longer result in failing healthchecks (#7523)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • JetStream shutdown signaling race fixed
    Affectsclustered JetStream

    During shutdown, when multiple goroutines listened on the cluster quit channel, another goroutine could consume the signal instead of monitorCluster(). The channel is now closed to signal all listeners, and shutdown waits on a second channel for the monitor to finish.

    - Fixed a race condition that could happen with shutdown signals when shutting down JetStream (#7536)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • Stream purge error-path deadlock fixed
    Affectsstream purges(Inferred)

    When consumer state did not match and writing consumer store state returned an error during a purge, the error path could deadlock by trying to reacquire the held stream lock. It now gets the stream name without reacquiring that lock.

    - Fixed a deadlock that could occur when purging a stream with mismatched consumer state (#7546)
    github.com/nats-io/nats-server/releases/tag/v2.11.11

Enhancements7

  • JetStream adds metalayer-log compaction thresholds

    To use Configure meta_compact, meta_compact_size, or both to use threshold-based compaction.

    - Added `meta_compact` and `meta_compact_size`, advanced JetStream config options to control how many log entries must be present in the metalayer log before snapshotting and compaction takes place (#7484, #7521)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • Configurable write_timeout behavior for clients, routes, gateways, and leafnodes
    - Added `write_timeout` option for clients, routes, gateways and leafnodes which controls the behaviour on reaching the `write_deadline`, values can be `default`, `retry` or `close` (#7513)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • JetStream meta-cluster snapshot statistics exposed in /jsz
    - Meta cluster snapshot statistics have been added to the `/jsz` endpoint (#7524)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • Direct-consumer details in /jsz

    To use Request /jsz with the direct-consumers=true query parameter.

    - The `/jsz` endpoint can now show direct consumers with the `direct-consumers?true` flag (#7543)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • Elapsed time added to JetStream catchup logs
    - Stream catchups will now use delete ranges more aggressively, speeding up catchups of large streams with many interior deletes (#7512)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • Subject-transform republishing accepts > in both fields

    To use A SubjectTransform mapping the stream’s sole subject, with > in both RePublish.Source and RePublish.Destination

    - Streams with subject transforms can now implicitly republish based on those transforms by configuring `>` for both republish source and destination (#7515)
    github.com/nats-io/nats-server/releases/tag/v2.11.11
  • JSZ reports Raft leader counts

    To use The raft flag must be set.

    - The `jsz` monitoring endpoint can now report leader counts (#7429)
    github.com/nats-io/nats-server/releases/tag/v2.11.11

Dependency updates6

    6 more, none mentioning security
    • - 1.25.4 (#7520)
    • - golang.org/x/crypto v0.43.0 (#7423)
    • - golang.org/x/sys v0.37.0 (#7423)
    • - golang.org/x/time v0.14.0 (#7423)
    • - golang.org/x/sys v0.38.0 (#7538)
    • - github.com/minio/highwayhash 070ab1a (#7539)
    Other changes16

    Summaries are produced by AI, so they can be wrong, and they are provided without warranty.