# NATS v2.11.11

2025-11-13 · [Original release notes](https://github.com/nats-io/nats-server/releases/tag/v2.11.11) · [Full summary](https://ratatosk.io/en/releases/nats/v2.11.11)

Important fixes 28 · Enhancements 7 · Dependency updates 6 · Other changes 16

## Check before upgrading

### Upgrade guide

- **2.11 upgrade guide for 2.10.x compatibility**
  > Refer to the [2.11 Upgrade Guide](https://docs.nats.io/release-notes/whats_new/whats_new_211) for backwards compatibility notes with 2.10.x.

  - Guides for upgrading between minor versions

## Important fixes (28)

### [Silent failure] JetStream stream catch-up health reporting fixed

- Affects: JetStream streams that are catching up

While a stream was catching up, its health check reported `group node unhealthy` because the Raft state did not advance until catch-up finished. The health check now checks for catch-up first and reports `stream catching up`.

> - Healthchecks now correctly report when streams are catching up, instead of showing them as unhealthy (#7535)

### JetStream filestore tombstone loss during secure erase fixed

- Affects: JetStream filestore setups using secure erase

When a message before tombstones in a block was erased, an ungraceful shutdown requiring an index rebuild could lose the tombstones and bring deleted messages back. The filestore now preserves those tombstones in this case.

> - The filestore no longer loses tombstones when using secure erase (#7384)

### JetStream filestore recovery sequence rollback fixed

- Affects: JetStream filestore setups recovering blocks containing only tombstones

After messages were acknowledged, their tombstones overflowed into new blocks, and the original message block was removed, an ungraceful restart could use the minimum known tombstone sequence and timestamp, rolling stream sequences back. With tombstones across multiple blocks, recovery could also assign an empty stream a first sequence that was too low; recovery now uses the maximum known tombstone sequence and timestamp in this case.

> - The filestore no longer loses the last sequence when recovering blocks containing only tombstones (#7384)

### Stream replica handling of out-of-order `SkipMsg` fixed

- Affects: streams with replicas

When a stream replica received an out-of-order `SkipMsg`, it could mark and delete a message that should not have been skipped, desynchronizing the stream. Skip sequences are now checked for ordering before they are applied, preventing this desynchronization during catchups.

> - Skipped message sequences are now checked for ordering before apply, fixing a potential stream desync on catchups (#7400)

### Filestore erased-message gap tracking fixed

- Affects: JetStream streams using the filestore (Inferred)

When a gap in filestore messages ended with an erased final message, the filestore failed to register the erased slots as deletes, causing constant cache invalidation, slow reads or catchup, and incorrect delete tracking. During state rebuilding, it now registers the missing sequence numbers as deletes.

> - Skipped message sequences now correctly detect gaps from erased message slots, fixing potential cache issues, slow reads and issues with catchups (#7399, #7401)

### Filestore sequence-gap deletion tracking fixed

- Affects: JetStream streams using the filestore

When a block with multiple `SkipMsgs` was empty at the start or fully empty, rebuilding could record sequence gaps in the `dmap` as deleted messages, even though the skips only advanced sequence numbers. Rebuilding now leaves those gaps out of the deleted-message map.

> - Skipped message sequences now correctly detect gaps from erased message slots, fixing potential cache issues, slow reads and issues with catchups (#7399, #7401)

### Raft peer activity reporting corrected

- Affects: Raft groups and peer-activity reporting in APIs, asset information, and monitoring endpoints

After leader changes or cluster resizes, old active timestamps could remain unchanged, making “last seen” and “active” values misleading. This release tracks peer activity more consistently and clears stale timestamps on leader changes, so reported values can differ from the previous release.

> - Raft groups now report peer activity more consistently, fixing some cases where asset info and monitoring endpoints may report misleading values after leader changes (#7402)

### Raft follower WAL truncation after catchup fixed

- Affects: Raft followers running parallel catchups

With multiple catchups running in parallel, a follower could accept a late append from another catchup after completion and truncate WAL entries already reported as eligible to commit. After catchup completes, unexpected catchup entries no longer truncate the WAL.

> - Raft groups will no longer permit truncations from unexpected catchup entries if the catchup is completed (#7424)

### NATS header suffix-matching bug fixed

- Affects: messages with header keys that share a common suffix

When getting, removing, or setting `Nats-Msg-Id` after `Previous-Nats-Msg-Id` was present, the operation could use the earlier key and value instead.

> - A couple of issues related to header handling have been fixed (#7465)

### `setHeader` now updates existing headers correctly

- Affects: code using `setHeader` to update existing headers

Previously, calling `setHeader` for `k` on an existing `k:v` header whose value was not prefixed with whitespace added another header instead of updating the existing one.

> - A couple of issues related to header handling have been fixed (#7465)

### JetStream empty-stream `NoWait` response fixed

- Affects: JetStream consumers using `NoWait` requests without expiry

When such a request reached the end of an empty stream without delivering messages, it did not receive `404 No Messages`; it now does.

> - No-wait requests now return a 400 No Messages response correctly if the stream is empty (#7466)

### Filestore tail hole-filling sequence error fixed

- Affects: JetStream filestore

When filling holes at the end of a block, an off-by-one error could mark one sequence as deleted and miss the last sequence.

> - An off-by-one when detecting holes at the end of a filestore block has been fixed (#7508)

### [Silent failure] False Raft quorum-loss reports fixed during leader shutdown and stepdown

- Affects: Raft groups led by the server

Previously, shutting down a leader with LDM or having it step down cleared peer timestamps, making its Raft groups report lost quorum, print “NO quorum, stalled,” and send an advisory. The leader now retains its peer timestamps after stepping down; timestamps for other followers can still be cleared once the new leader comes online.

> - Raft groups should no longer report no quorum incorrectly when shutting down (#7522)

### JetStream filestore end-of-block hole detection fixed

- Affects: JetStream filestore (Inferred)

When detecting holes at a block's end, a tombstone could be treated as the last sequence, causing deletes to be missed. The filestore now bases sequence handling on the last known-good sequence.

> - An off-by-one when detecting holes at the end of a filestore block has been fixed (#7525)

### NATS server prefers binary snapshots on new route connections

- Affects: nodes on new route connections to servers running v2.10.0 or later

Before `STATSZ` arrived on a newly established route, the server could fall back to the legacy format, whose `[]uint64` representation of interior deletes could use disastrous amounts of memory for streams with many interior deletes.

> - Binary stream snapshots are now preferred by default for nodes on new route connections (#7479)

### Interest-based stream replicas recover from desynchronization

- Affects: clustered JetStream streams using `InterestPolicy`

When consumer changes establish interest at different logical times across servers, replicas could desynchronize and store different initial messages. Servers can now detect this `noInterest` desynchronization and remove leading messages with no interest, though the underlying ordering race can still occur.

> - Improve interest detection when consumers are created or deleted across different servers (#7440)

### Per-gateway `hop` header corruption fixed

- Affects: setups using message tracing across gateways

With message tracing enabled, setting the `hop` header for each gateway could corrupt headers. The header is now set per gateway without the described corruption.

> - When using message tracing, header corruption when setting the hop header has been fixed (#7443)

### Lame-duck connection-limit errors fixed

- Affects: accounts with a maximum-connections limit on servers using lame-duck mode

When account connections were near their limit during lame-duck mode, closed connections remained in the count and reconnecting clients were counted twice, so clients not yet reconnected could receive a `maximum account active connections exceeded` error. Connection accounting now avoids that double-counting; during shutdown with `s.Shutdown()`, remaining clients can still be closed without accounting.

> - Shutting down a server using lame-duck mode should no longer result in max connection exceeded errors (#7527)

### JetStream API handler races and nil-meta panics fixed

- Affects: JetStream API handlers, including consumer-info handling

Some handlers accessed `cc.meta` without locking it, creating race conditions, and could panic when it was nil. The affected handlers no longer access it unlocked, and the consumer-info handler checks for nil before use.

> - Race conditions and potential panics fixed in the handling of some JetStream API handlers (#7380)

### Filestore TTL and scheduling state writes honor `sync_always`

- Affects: JetStream filestores using `sync_always` for TTL or scheduling state files

> - The filestore now correctly obeys `sync_always` for writing TTL and scheduling state files (#7385)

### Filestore locks released on message-erasure errors

- Affects: JetStream streams using the filestore (Inferred)

If `slotInfo` or `mb.eraseMsg` returned an error during message erasure, the filestore could return without releasing the relevant locks; it now releases them before returning the error.

> - The filestore will now correctly release locks when erasing messages returns an error (#7431)

### JetStream filestore shutdown race fixed

- Affects: JetStream filestore

During shutdown, if the lock was re-acquired while `index.db` was being generated, `fs.StoreMsg` could proceed after the index was written, leaving an invalid index on restart.

> - Fixed a race condition in the filestore that could happen between storing messages and shutting down (#7496)

### Parallel stream restoration panic fixed

- Affects: parallel stream restoration during startup

During startup, `ats.Register()` could run before the access-time service had registered a value, causing an `access time service not running` panic. `AccessTime()` now stores and returns an initial timestamp in this case, which may be slightly stale on startup.

> - A panic that could occur when recovering streams in parallel has been fixed (#7503)

### JetStream monitors use the correct Raft group after stream recreation

- Affects: JetStream metalayer recovery with recreated streams and associated consumers (Inferred)

When a stream and consumer were in a meta snapshot and the stream was deleted and recreated with a different config, their monitor goroutines could use a Raft node from the deleted group and fail to function properly.

> - Fixed a bug on metalayer recovery where stream and consumer monitor goroutines for recreated assets would run with the wrong Raft group (#7510)

### Snapshot installation fixed during R1-to-R3 stream scale-up

- Affects: JetStream streams scaled up from R1 to R3

If JetStream catchup failed partway during a scale-up, incomplete peers could try to become leader and desynchronize the stream. The snapshot is now installed so recovery can process it, and new peers cannot become leader until they are fully synced.

> - Scaling up an asset from R1 now results in an installed snapshot, allowing recovery after restart if interrupted, avoiding a potential desync (#7509)

### JetStream recovery health checks no longer include deleted consumers

- Affects: JetStream metalayer snapshot recovery

When a consumer in the meta snapshot was deleted during recovery, it could reappear in the health check and cause a `consumer not found` error.

> - Consumers that existed in a metalayer snapshot but were deleted on recovery will no longer result in failing healthchecks (#7523)

### JetStream shutdown signaling race fixed

- Affects: clustered JetStream

During shutdown, when multiple goroutines listened on the cluster quit channel, another goroutine could consume the signal instead of `monitorCluster()`. The channel is now closed to signal all listeners, and shutdown waits on a second channel for the monitor to finish.

> - Fixed a race condition that could happen with shutdown signals when shutting down JetStream (#7536)

### Stream purge error-path deadlock fixed

- Affects: stream purges (Inferred)

When consumer state did not match and writing consumer store state returned an error during a purge, the error path could deadlock by trying to reacquire the held stream lock. It now gets the stream name without reacquiring that lock.

> - Fixed a deadlock that could occur when purging a stream with mismatched consumer state (#7546)

## Enhancements (7)

- JetStream adds metalayer-log compaction thresholds (To use: Configure `meta_compact`, `meta_compact_size`, or both to use threshold-based compaction.)
- Configurable `write_timeout` behavior for clients, routes, gateways, and leafnodes
- JetStream meta-cluster snapshot statistics exposed in `/jsz`
- Direct-consumer details in `/jsz` (To use: Request `/jsz` with the `direct-consumers=true` query parameter.)
- Elapsed time added to JetStream catchup logs
- Subject-transform republishing accepts `>` in both fields (To use: A `SubjectTransform` mapping the stream’s sole subject, with `>` in both `RePublish.Source` and `RePublish.Destination`)
- JSZ reports Raft leader counts (To use: The `raft` flag must be set.)

## Dependency updates (6)

6 more, none mentioning security

- `- 1.25.4 (#7520)`
- `- golang.org/x/crypto v0.43.0 (#7423)`
- `- golang.org/x/sys v0.37.0 (#7423)`
- `- golang.org/x/time v0.14.0 (#7423)`
- `- golang.org/x/sys v0.38.0 (#7538)`
- `- github.com/minio/highwayhash 070ab1a (#7539)`

## Other changes (16)

- Reduced allocations in sublist and subject transforms
- Observer Mode first-start logging clarified
- Bulk removal of eligible JetStream filestore message blocks
- Snapshot attempts limited to server shutdown
- Parallel stream loading during JetStream startup
- JetStream consumers start immediately after their parent streams
- Catchup selects `LoadNextMsg` at lower delete thresholds
- Stream leader catchup race during scale-up fixed
- JetStream startup and recovery concurrency capped at `len(dios)`
- Hash checks switched to `highwayhash.Digest64`
- JetStream filestore cleanup of empty leading tombstone blocks
- Mirror consumer setup data race fixed
- JetStream filestore cache expiry fixed for repeated last-sequence reads
- NATS header handling avoids `key+":"` string allocations
- Raft recovery waits for no-op application before reporting leadership
- JetStream filestore keeps the message block mutex held during skip writes

---

Summaries are produced by AI, so they can be wrong, and they are provided without warranty.
