NATSv2.11.11
Important fixes 28Enhancements 7Dependency updates 6Other changes 16
Email is sent weekly only when there are changes to check; critical security advisories are sent immediately.
Check before upgrading
Upgrade guide1
2.11 upgrade guide for 2.10.x compatibility
- Guides for upgrading between minor versions
Refer to the [2.11 Upgrade Guide](https://docs.nats.io/release-notes/whats_new/whats_new_211) for backwards compatibility notes with 2.10.x.github.com/nats-io/nats-server/releases/tag/v2.11.11
Important fixes28Silent failure2
JetStream stream catch-up health reporting fixedSilent failure
AffectsJetStream streams that are catching upWhile a stream was catching up, its health check reported
group node unhealthybecause the Raft state did not advance until catch-up finished. The health check now checks for catch-up first and reportsstream catching up.- Healthchecks now correctly report when streams are catching up, instead of showing them as unhealthy (#7535)
github.com/nats-io/nats-server/releases/tag/v2.11.11JetStream filestore tombstone loss during secure erase fixed
AffectsJetStream filestore setups using secure eraseWhen a message before tombstones in a block was erased, an ungraceful shutdown requiring an index rebuild could lose the tombstones and bring deleted messages back. The filestore now preserves those tombstones in this case.
- The filestore no longer loses tombstones when using secure erase (#7384)
github.com/nats-io/nats-server/releases/tag/v2.11.11JetStream filestore recovery sequence rollback fixed
AffectsJetStream filestore setups recovering blocks containing only tombstonesAfter messages were acknowledged, their tombstones overflowed into new blocks, and the original message block was removed, an ungraceful restart could use the minimum known tombstone sequence and timestamp, rolling stream sequences back. With tombstones across multiple blocks, recovery could also assign an empty stream a first sequence that was too low; recovery now uses the maximum known tombstone sequence and timestamp in this case.
- The filestore no longer loses the last sequence when recovering blocks containing only tombstones (#7384)
github.com/nats-io/nats-server/releases/tag/v2.11.11Stream replica handling of out-of-order
SkipMsgfixedAffectsstreams with replicasWhen a stream replica received an out-of-order
SkipMsg, it could mark and delete a message that should not have been skipped, desynchronizing the stream. Skip sequences are now checked for ordering before they are applied, preventing this desynchronization during catchups.- Skipped message sequences are now checked for ordering before apply, fixing a potential stream desync on catchups (#7400)
github.com/nats-io/nats-server/releases/tag/v2.11.11Filestore erased-message gap tracking fixed
AffectsJetStream streams using the filestore(Inferred)When a gap in filestore messages ended with an erased final message, the filestore failed to register the erased slots as deletes, causing constant cache invalidation, slow reads or catchup, and incorrect delete tracking. During state rebuilding, it now registers the missing sequence numbers as deletes.
- Skipped message sequences now correctly detect gaps from erased message slots, fixing potential cache issues, slow reads and issues with catchups (#7399, #7401)
github.com/nats-io/nats-server/releases/tag/v2.11.11Filestore sequence-gap deletion tracking fixed
AffectsJetStream streams using the filestoreWhen a block with multiple
SkipMsgswas empty at the start or fully empty, rebuilding could record sequence gaps in thedmapas deleted messages, even though the skips only advanced sequence numbers. Rebuilding now leaves those gaps out of the deleted-message map.- Skipped message sequences now correctly detect gaps from erased message slots, fixing potential cache issues, slow reads and issues with catchups (#7399, #7401)
github.com/nats-io/nats-server/releases/tag/v2.11.11Raft peer activity reporting corrected
AffectsRaft groups and peer-activity reporting in APIs, asset information, and monitoring endpointsAfter leader changes or cluster resizes, old active timestamps could remain unchanged, making “last seen” and “active” values misleading. This release tracks peer activity more consistently and clears stale timestamps on leader changes, so reported values can differ from the previous release.
- Raft groups now report peer activity more consistently, fixing some cases where asset info and monitoring endpoints may report misleading values after leader changes (#7402)
github.com/nats-io/nats-server/releases/tag/v2.11.11Raft follower WAL truncation after catchup fixed
AffectsRaft followers running parallel catchupsWith multiple catchups running in parallel, a follower could accept a late append from another catchup after completion and truncate WAL entries already reported as eligible to commit. After catchup completes, unexpected catchup entries no longer truncate the WAL.
- Raft groups will no longer permit truncations from unexpected catchup entries if the catchup is completed (#7424)
github.com/nats-io/nats-server/releases/tag/v2.11.11NATS header suffix-matching bug fixed
Affectsmessages with header keys that share a common suffixWhen getting, removing, or setting
Nats-Msg-IdafterPrevious-Nats-Msg-Idwas present, the operation could use the earlier key and value instead.- A couple of issues related to header handling have been fixed (#7465)
github.com/nats-io/nats-server/releases/tag/v2.11.11setHeadernow updates existing headers correctlyAffectscode usingsetHeaderto update existing headersPreviously, calling
setHeaderforkon an existingk:vheader whose value was not prefixed with whitespace added another header instead of updating the existing one.- A couple of issues related to header handling have been fixed (#7465)
github.com/nats-io/nats-server/releases/tag/v2.11.11JetStream empty-stream
NoWaitresponse fixedAffectsJetStream consumers usingNoWaitrequests without expiryWhen such a request reached the end of an empty stream without delivering messages, it did not receive
404 No Messages; it now does.- No-wait requests now return a 400 No Messages response correctly if the stream is empty (#7466)
github.com/nats-io/nats-server/releases/tag/v2.11.11Filestore tail hole-filling sequence error fixed
AffectsJetStream filestoreWhen filling holes at the end of a block, an off-by-one error could mark one sequence as deleted and miss the last sequence.
- An off-by-one when detecting holes at the end of a filestore block has been fixed (#7508)
github.com/nats-io/nats-server/releases/tag/v2.11.11False Raft quorum-loss reports fixed during leader shutdown and stepdownSilent failure
AffectsRaft groups led by the serverPreviously, shutting down a leader with LDM or having it step down cleared peer timestamps, making its Raft groups report lost quorum, print “NO quorum, stalled,” and send an advisory. The leader now retains its peer timestamps after stepping down; timestamps for other followers can still be cleared once the new leader comes online.
- Raft groups should no longer report no quorum incorrectly when shutting down (#7522)
github.com/nats-io/nats-server/releases/tag/v2.11.11JetStream filestore end-of-block hole detection fixed
AffectsJetStream filestore(Inferred)When detecting holes at a block's end, a tombstone could be treated as the last sequence, causing deletes to be missed. The filestore now bases sequence handling on the last known-good sequence.
- An off-by-one when detecting holes at the end of a filestore block has been fixed (#7525)
github.com/nats-io/nats-server/releases/tag/v2.11.11NATS server prefers binary snapshots on new route connections
Affectsnodes on new route connections to servers running v2.10.0 or laterBefore
STATSZarrived on a newly established route, the server could fall back to the legacy format, whose[]uint64representation of interior deletes could use disastrous amounts of memory for streams with many interior deletes.- Binary stream snapshots are now preferred by default for nodes on new route connections (#7479)
github.com/nats-io/nats-server/releases/tag/v2.11.11Interest-based stream replicas recover from desynchronization
Affectsclustered JetStream streams usingInterestPolicyWhen consumer changes establish interest at different logical times across servers, replicas could desynchronize and store different initial messages. Servers can now detect this
noInterestdesynchronization and remove leading messages with no interest, though the underlying ordering race can still occur.- Improve interest detection when consumers are created or deleted across different servers (#7440)
github.com/nats-io/nats-server/releases/tag/v2.11.11Per-gateway
hopheader corruption fixedAffectssetups using message tracing across gatewaysWith message tracing enabled, setting the
hopheader for each gateway could corrupt headers. The header is now set per gateway without the described corruption.- When using message tracing, header corruption when setting the hop header has been fixed (#7443)
github.com/nats-io/nats-server/releases/tag/v2.11.11Lame-duck connection-limit errors fixed
Affectsaccounts with a maximum-connections limit on servers using lame-duck modeWhen account connections were near their limit during lame-duck mode, closed connections remained in the count and reconnecting clients were counted twice, so clients not yet reconnected could receive a
maximum account active connections exceedederror. Connection accounting now avoids that double-counting; during shutdown withs.Shutdown(), remaining clients can still be closed without accounting.- Shutting down a server using lame-duck mode should no longer result in max connection exceeded errors (#7527)
github.com/nats-io/nats-server/releases/tag/v2.11.11JetStream API handler races and nil-meta panics fixed
AffectsJetStream API handlers, including consumer-info handlingSome handlers accessed
cc.metawithout locking it, creating race conditions, and could panic when it was nil. The affected handlers no longer access it unlocked, and the consumer-info handler checks for nil before use.- Race conditions and potential panics fixed in the handling of some JetStream API handlers (#7380)
github.com/nats-io/nats-server/releases/tag/v2.11.11Filestore TTL and scheduling state writes honor
sync_alwaysAffectsJetStream filestores usingsync_alwaysfor TTL or scheduling state files- The filestore now correctly obeys `sync_always` for writing TTL and scheduling state files (#7385)
github.com/nats-io/nats-server/releases/tag/v2.11.11Filestore locks released on message-erasure errors
AffectsJetStream streams using the filestore(Inferred)If
slotInfoormb.eraseMsgreturned an error during message erasure, the filestore could return without releasing the relevant locks; it now releases them before returning the error.- The filestore will now correctly release locks when erasing messages returns an error (#7431)
github.com/nats-io/nats-server/releases/tag/v2.11.11JetStream filestore shutdown race fixed
AffectsJetStream filestoreDuring shutdown, if the lock was re-acquired while
index.dbwas being generated,fs.StoreMsgcould proceed after the index was written, leaving an invalid index on restart.- Fixed a race condition in the filestore that could happen between storing messages and shutting down (#7496)
github.com/nats-io/nats-server/releases/tag/v2.11.11Parallel stream restoration panic fixed
Affectsparallel stream restoration during startupDuring startup,
ats.Register()could run before the access-time service had registered a value, causing anaccess time service not runningpanic.AccessTime()now stores and returns an initial timestamp in this case, which may be slightly stale on startup.- A panic that could occur when recovering streams in parallel has been fixed (#7503)
github.com/nats-io/nats-server/releases/tag/v2.11.11JetStream monitors use the correct Raft group after stream recreation
AffectsJetStream metalayer recovery with recreated streams and associated consumers(Inferred)When a stream and consumer were in a meta snapshot and the stream was deleted and recreated with a different config, their monitor goroutines could use a Raft node from the deleted group and fail to function properly.
- Fixed a bug on metalayer recovery where stream and consumer monitor goroutines for recreated assets would run with the wrong Raft group (#7510)
github.com/nats-io/nats-server/releases/tag/v2.11.11Snapshot installation fixed during R1-to-R3 stream scale-up
AffectsJetStream streams scaled up from R1 to R3If JetStream catchup failed partway during a scale-up, incomplete peers could try to become leader and desynchronize the stream. The snapshot is now installed so recovery can process it, and new peers cannot become leader until they are fully synced.
- Scaling up an asset from R1 now results in an installed snapshot, allowing recovery after restart if interrupted, avoiding a potential desync (#7509)
github.com/nats-io/nats-server/releases/tag/v2.11.11JetStream recovery health checks no longer include deleted consumers
AffectsJetStream metalayer snapshot recoveryWhen a consumer in the meta snapshot was deleted during recovery, it could reappear in the health check and cause a
consumer not founderror.- Consumers that existed in a metalayer snapshot but were deleted on recovery will no longer result in failing healthchecks (#7523)
github.com/nats-io/nats-server/releases/tag/v2.11.11JetStream shutdown signaling race fixed
Affectsclustered JetStreamDuring shutdown, when multiple goroutines listened on the cluster quit channel, another goroutine could consume the signal instead of
monitorCluster(). The channel is now closed to signal all listeners, and shutdown waits on a second channel for the monitor to finish.- Fixed a race condition that could happen with shutdown signals when shutting down JetStream (#7536)
github.com/nats-io/nats-server/releases/tag/v2.11.11Stream purge error-path deadlock fixed
Affectsstream purges(Inferred)When consumer state did not match and writing consumer store state returned an error during a purge, the error path could deadlock by trying to reacquire the held stream lock. It now gets the stream name without reacquiring that lock.
- Fixed a deadlock that could occur when purging a stream with mismatched consumer state (#7546)
github.com/nats-io/nats-server/releases/tag/v2.11.11
Enhancements7
JetStream adds metalayer-log compaction thresholds
To use Configure
meta_compact,meta_compact_size, or both to use threshold-based compaction.- Added `meta_compact` and `meta_compact_size`, advanced JetStream config options to control how many log entries must be present in the metalayer log before snapshotting and compaction takes place (#7484, #7521)
github.com/nats-io/nats-server/releases/tag/v2.11.11Configurable
write_timeoutbehavior for clients, routes, gateways, and leafnodes- Added `write_timeout` option for clients, routes, gateways and leafnodes which controls the behaviour on reaching the `write_deadline`, values can be `default`, `retry` or `close` (#7513)
github.com/nats-io/nats-server/releases/tag/v2.11.11JetStream meta-cluster snapshot statistics exposed in
/jsz- Meta cluster snapshot statistics have been added to the `/jsz` endpoint (#7524)
github.com/nats-io/nats-server/releases/tag/v2.11.11Direct-consumer details in
/jszTo use Request
/jszwith thedirect-consumers=truequery parameter.- The `/jsz` endpoint can now show direct consumers with the `direct-consumers?true` flag (#7543)
github.com/nats-io/nats-server/releases/tag/v2.11.11Elapsed time added to JetStream catchup logs
- Stream catchups will now use delete ranges more aggressively, speeding up catchups of large streams with many interior deletes (#7512)
github.com/nats-io/nats-server/releases/tag/v2.11.11Subject-transform republishing accepts
>in both fieldsTo use A
SubjectTransformmapping the stream’s sole subject, with>in bothRePublish.SourceandRePublish.Destination- Streams with subject transforms can now implicitly republish based on those transforms by configuring `>` for both republish source and destination (#7515)
github.com/nats-io/nats-server/releases/tag/v2.11.11JSZ reports Raft leader counts
To use The
raftflag must be set.- The `jsz` monitoring endpoint can now report leader counts (#7429)
github.com/nats-io/nats-server/releases/tag/v2.11.11
Dependency updates6
6 more, none mentioning security
- - 1.25.4 (#7520)
- - golang.org/x/crypto v0.43.0 (#7423)
- - golang.org/x/sys v0.37.0 (#7423)
- - golang.org/x/time v0.14.0 (#7423)
- - golang.org/x/sys v0.38.0 (#7538)
- - github.com/minio/highwayhash 070ab1a (#7539)
Other changes16
Reduced allocations in sublist and subject transforms
- Reduced allocations in the sublist and subject transforms (#7519)
github.com/nats-io/nats-server/releases/tag/v2.11.11Observer Mode first-start logging clarified
- Improved the logging for observer mode (#7433)
github.com/nats-io/nats-server/releases/tag/v2.11.11Bulk removal of eligible JetStream filestore message blocks
- Improved the performance of enforcing `max_bytes` and `max_msgs` limits (#7455)
github.com/nats-io/nats-server/releases/tag/v2.11.11Snapshot attempts limited to server shutdown
- Streams and consumers will no longer unnecessarily snapshot when being removed or scaling down (#7495)
github.com/nats-io/nats-server/releases/tag/v2.11.11Parallel stream loading during JetStream startup
- Streams are now loaded in parallel when enabling JetStream, often reducing the time it takes to start up the server (#7482)
github.com/nats-io/nats-server/releases/tag/v2.11.11JetStream consumers start immediately after their parent streams
- Streams are now loaded in parallel when enabling JetStream, often reducing the time it takes to start up the server (#7482)
github.com/nats-io/nats-server/releases/tag/v2.11.11Catchup selects
LoadNextMsgat lower delete thresholds- Stream catchups will now use delete ranges more aggressively, speeding up catchups of large streams with many interior deletes (#7512)
github.com/nats-io/nats-server/releases/tag/v2.11.11Stream leader catchup race during scale-up fixed
- A race condition where subscriptions may not be set up before catchup requests are sent after a leader change has been fixed (#7518)
github.com/nats-io/nats-server/releases/tag/v2.11.11JetStream startup and recovery concurrency capped at
len(dios)- JetStream recovery parallelism now matches the I/O gated semaphore (#7526)
github.com/nats-io/nats-server/releases/tag/v2.11.11Hash checks switched to
highwayhash.Digest64- Reduced heap allocations in hash checks (#7539)
github.com/nats-io/nats-server/releases/tag/v2.11.11JetStream filestore cleanup of empty leading tombstone blocks
- The filestore now correctly cleans up empty blocks when selecting the next first block (#7384)
github.com/nats-io/nats-server/releases/tag/v2.11.11Mirror consumer setup data race fixed
- Fixed a data race on a wait group when mirroring streams (#7395)
github.com/nats-io/nats-server/releases/tag/v2.11.11JetStream filestore cache expiry fixed for repeated last-sequence reads
- Caches will now no longer expire unnecessarily when re-reading the same sequences multiple times in first-matching code paths (#7435)
github.com/nats-io/nats-server/releases/tag/v2.11.11NATS header handling avoids
key+":"string allocations- A couple of issues related to header handling have been fixed (#7465)
github.com/nats-io/nats-server/releases/tag/v2.11.11Raft recovery waits for no-op application before reporting leadership
- Raft groups will now only report leadership status after a no-op entry on recovery (#7460)
github.com/nats-io/nats-server/releases/tag/v2.11.11JetStream filestore keeps the message block mutex held during skip writes
- Writing skip message records in the filestore no longer releases and reacquires the lock unnecessarily (#7508)
github.com/nats-io/nats-server/releases/tag/v2.11.11
Summaries are produced by AI, so they can be wrong, and they are provided without warranty.