Longhorn
v1.12.2Storage & DataA stability-focused maintenance release addresses data-loss risks, panics, deadlocks, mount delays, device-state handling, and defects across both data engines. It also improves networking and Helm charts, with upgrade prerequisites, V2 operational constraints, and secret-sanitization fixes included.
Action needed (1)
securityCSI request-log secret sanitization
CSI method-level request logs no longer bypass secret sanitization. The fix is included in v1.12.2.
Check if affected (4)
breakingV2 Data Engine live-upgrade constraint
Applies if you perform a live upgrade of the V2 Data Engine from v1.12.2 to v1.13.0.
Live upgrade of the V2 Data Engine from v1.12.2 to v1.13.0 is not supported in this release.
breakingLegacy linked-clone volume operation restriction
Applies if you upgrade to v1.12.1 or later.
After upgrading to v1.12.1 or later,
legacy linked-clone volumescan only be detached or deleted. This restriction ships in v1.12.2.breakingV2 Data Engine CPU requirement
Applies if you configure fewer than 2 CPU cores for the V2 Data Engine.
The V2 Data Engine should be configured with at least 2 CPU cores for stability. With a multi-core mask, one core is reserved for the admin reactor, leaving one fewer core for I/O in v1.12.2.
breaking
Kubernetesv1.25 installation prerequisiteApplies if you install Longhorn v1.12.2 on a cluster running
Kubernetesearlier than v1.25.Longhorn v1.12.2 requires the cluster to run
Kubernetesv1.25 or later before installation.
All 74 other recorded changesfixes 60 · value changes 8 · additions 4 · constraints 2
fixes (60)
- Fixes a bug in which acknowledged writes could be silently lost during a replica rebuild.
- Fixes replica rebuilds that randomly failed on busy networks because a single lost TCP keepalive terminated the idle sync-agent gRPC connection after 20 seconds.
- the rebuild switched the parent of the destination replica head while it was still serving I/O
spdk_tgtcould abort withdouble free or corruption (fasttop)during replica rebuilds.- Large reads (for example, 1 MiB at a high queue depth) could flood the NVMe-oF TCP target with flush retries
- Deleting a snapshot that is smaller than its child and not a multiple of 512 MiB could leave lvol metadata that SPDK fails to load
- Fixes a critical issue in which a failure while setting up a rebuild on the destination replica faulted the entire engine
- Fixes a lost wakeup in the SPDK erasure-coding bdev that could stall writes and leave the SPDK target unresponsive
spdk_tgtcrashes with "double free or corruption (fasttop)" around replica rebuilds- Concurrent unaligned writes during replica rebuild lose data
- v2 instance-manager is crashed by disk recovery failure in SPDK
- a rebuilding replica sometimes disappears in the
Engine.Status. ReplicaModeMap - Concurrent volume attachments can create multiple NVMe-oF transports and crash spdk_tgt
- V2 instance manager live upgrade stuck in relocating-engines
- V2 instance-manager
spdk_tgtallocates its hugepages from another pod's hugetlbfs mount - A stalled NVMe device makes engine frontends of unrelated v2 volumes fail validation and be recreated
- nvme list zombie processes leak when NVMe admin queue returns EAGAIN
- SystemBackup stuck in
CreatingVolumeBackups - v2 spdk_tgt aborts with assertion in blob_insert_cluster_msg due to an extent page race
- Fast replica rebuild skips a snapshot file that was changed by purge/coalesce
- Transient status RPC failure marks healthy replica failed and can fault a volume
- Disk health data never collected for V1 filesystem disks whose path is not itself a mount point
- Force-deleted pod VolumeAttachment cleanup ignores "Waiting for detach" events
- csi-resizer fails to start with "unrecognized feature gate: RecoverVolumeExpansionFailure"
- V2 DR volume restore does not retry when backupstore lock is held
- All v1 volume I/O stalls silently with longhorn-engine v1.7.1 through v1.9.2 on Rocky Linux 9.7 / RKE2
- Sharded/EC volume: writes stall to ~0 and wedge the SPDK target on arm64/AIO
- Volume expansion snapshot CR name expand-<size> collides across volumes
- Longhorn v2 engine crashes on large-block reads in interrupt mode
- v2 volume gets stuck in
Attachingin an IPv6 storage network environment - v1.11.3 regenerates the webhook CA on upgrade from an ECDSA (dynamiclistener) CA, deadlocking every longhorn-manager
- SystemBackup fails when a snapshot has an empty status.creationTime
- invalid block disks should not block Longhorn system uninstallation
- deleting nodes.longhorn.io failed due to KubernetesNodeGone
- v1.12.1 chart: new default-on
networkPolicies.silently breaks Prometheus scraping of longhorn-managerrestrictInternalTraffic - V2 replica rebuild destination failure faults the entire engine
- NodeStageVolume can trigger destructive mkfs on existing volume during temporary I/O errors, risking data loss
- instance-manager crashes with "concurrent map iteration and map write" in ProcessList
- SystemRestore job ignores the registry-secret setting
- DR replicas break
- Cloned volume stuck "not ready for workloads" after the clone finishes
- Runtime container images contain development (-devel) packages
- Adding V2 disk using the "Default Data Path" /dev/disk/by-id/scsi-* path fails
- Request Schema in Upgrade Responder is Outdate
- v2 volume expansion is reported complete before the frontend dm device is resized
- Terminal workload pods cause NodePublishVolume to abort with "no Pending workload pods"
- v1 engine never starts: a stale
stoppedprocess record makes createInstance a silent no-op - v1.12.1 default NetworkPolicies miss legitimate non-pod-labeled and host-originated callers, silently breaking CSI attach/detach
- Recurring-job pods report success after startup or volume execution errors
- V1 volumes not rebuilding after cluster shutdown
- v2: spdk_tgt aborts on bdev_channel_destroy_resource assertion when rebuild finish sets the dst replica parent under I/O
- V2 instance-manager liveness probe errors ('test: -eq: unary operator expected') -> self-kill -> node-wide replica fault under rebuild load
- V2 Instance Manager panics in backupstore when S3 volume.cfg is missing, taking all node volumes offline
- Admission webhook calls for one resource type (backups.longhorn.io) fail with instant EOF
- Recurring snapshot-cleanup aborts the entire run when any volume's replica is rebuilding
- - [BACKPORT][v1.12.2][BUG] V2 backup readers race snapshot cleanup and panic the instance manager [14021](https://github.com/longhorn/longhorn/issues/14021) - @yangchiu
- - [BACKPORT][v1.12.2][BUG] Live migration and replica rebuild enter circular wait after nodes are uncordoned [14154](https://github.com/longhorn/longhorn/issues/14154) - @chriscchien @carterli0407-cell
- - [BACKPORT][v1.12.2][BUG] instance-manager panics (nil pointer) in sync.(*Task).HashSnapshotStatus when a replica is rebuilding: shadowed err in isRebuilding branch [14252](https://github.com/longhorn/longhorn/issues/14252) - @chriscchien
- - [BACKPORT][v1.12.2][BUG] RWX volume mounts are delayed up to 2 minutes when the share manager is stopped and started repeatedly (e.g. by a CronJob) [14178](https://github.com/longhorn/longhorn/issues/14178) - @roger-ryao
- - [BACKPORT][v1.12.2][BUG] Longhorn block disk remains Ready=False and Schedulable=False when NVMe device is bound to vfio-pci [13934](https://github.com/longhorn/longhorn/issues/13934) - @derekbit @chriscchien
value changes (8)
- Skip the filesystem sync when suspending the frontend for replica rebuilding
- Sync v2 volume and engine image fields after upgrade
- CSI ignores EC StorageClass parameters (silent replicated fallback)
- CSI Resize may idle for a long time after volume clone or restore
- (chart) Precedence of image.*.*.registry
- nsmounter: kubelet PID lookup on Talos takes seconds per call
- - [BACKPORT][v1.12.2][IMPROVEMENT] Avoid re-sending an unchanged Node status update [14145](https://github.com/longhorn/longhorn/issues/14145) - @hookak @chriscchien
- - [BACKPORT][v1.12.2][TASK] Add Helm chart image HIP-0015 annotations [14204](https://github.com/longhorn/longhorn/issues/14204) - @roger-ryao @carterli0407-cell
additions (4)
- You can now limit the number of I/O queues the kernel NVMe-TCP initiator creates for V2 volumes
- Enable CPU Isolation for V2 Data Engine
- backup target availability status metric
- - [BACKPORT][v1.12.2][IMPROVEMENT] Chart value to allow metrics scrapers through the longhorn-manager NetworkPolicy [13989](https://github.com/longhorn/longhorn/issues/13989) - @yangchiu
constraints (2)
- Previously they always used IPv4.
- Longhorn may not reject expansion requests when the volume clone is in-progress
A weekly email arrives when a release needs action. Like the security patches and breaking changes in this release.