Longhorn
v1.13.0Storage & DataA feature-rich release expands V2 Data Engine, scheduling, snapshots, metrics, and operational capabilities while fixing volume, backup, networking, resilience, and stability issues. Upgrade prerequisites need review, and the release includes a security fix plus opt-in resilience and CPU-management improvements that leave existing defaults unchanged.
Action needed (1)
securityCSI method-level request log secret sanitization
CSI method-level request logs no longer bypass secret sanitization. The fix is tracked in issue
13613.
Check if affected (5)
breakingKubernetes version requirement for the
CSI external-provisionerApplies when installing or upgrading to Longhorn
v1..13. 0 The
CSI external-provisioneris upgraded tov6., so clusters must run Kubernetes3. 0 v1.or later before installing or upgrading to Longhorn34 v1..13. 0 breakingRestrictions on older
V2 linked-clone volumesApplies if you use
V2 linked-clone volumescreated in Longhornv1.or earlier.12. 0 V2 linked-clone volumescreated in Longhornv1.or earlier can only be detached or deleted.12. 0 breakingV2 live upgrade support for Longhorn
v1.13. 0 Applicability is not stated in the release notes.
V2 live upgrade to Longhorn
v1.is supported only from Longhorn13. 0 v1.. Upgrades from12. 2 v1.or12. 0 v1., or upgrades that do not meet the documented prerequisites, require all V2 volumes to be detached and their replicas stopped first.12. 1 breakingThree-replica
longhorn-global-managerDeploymentApplicability is not stated in the release notes.
The
longhorn-global-managerDeployment is created with three replicas by default during installation and upgrade. At least one pod must be schedulable before an upgrade.breakingmTLS coverage for instance manager gRPC services
Applies if the
longhorn-grpc-tlssecret is configured.mTLS covers all instance manager gRPC services, and every gRPC port requires a valid client certificate when the
longhorn-grpc-tlssecret is configured.
All 118 other recorded changesfixes 78 · additions 28 · value changes 9 · constraints 2 · defaults 1
fixes (78)
- Longhorn v1.13.0 fixes a V2 linked-clone backup restore issue that could produce a corrupted volume when the source volume or the snapshot the clone was created from no longer existed.
- Longhorn v1.13.0 fixes a CSI volume clone issue where cloning a volume with
dataLocality: strict-localcould fail withhard affinity cannot be satisfied. - Longhorn v1.13.0 fixes a node cleanup issue where a Longhorn node could not be removed after its Kubernetes node had been deleted without eviction.
- Longhorn v1.13.0 fixes a backing image issue where copies were always transferred over IPv4, so on IPv6 single-stack and IPv6-first dual-stack clusters a backing image stayed at one copy.
- [IMPROVEMENT] [CLI] preflight checks on Elemental3-based systems fails
- [BUG] Runtime container images contain development (-devel) packages
- [BUG] DR replicas break
- [BUG] Longhorn may not reject expansion requests when the volume clone is in-progress
- [BUG] Request Schema in Upgrade Responder is Outdate
- [BUG]
Disconnect Volume Node Network For More Than Pod Eviction Timeout While Workload Heavy Writing With RWX Fast Failover Disabledfails on v2 data engine - [BUG] deleting nodes.longhorn.io failed due to KubernetesNodeGone
- [BUG] Cloned volume stuck "not ready for workloads" after the clone finishes (v1.12.0, v1 engine)
- [BUG] SystemRestore job ignores the registry-secret setting
- [BUG] After host restart one node no longer abe to start any replica
- [BUG] v2 workload IO gets stuck in
Continuous IO Test - [BUG] Terminal workload pods cause NodePublishVolume to abort with "no Pending workload pods"
- [BUG] v2 volumes might get stuck in
deletingstate - [UI][BUG] The restore of V2 Linked Clone volumes misses src volume handling
- [BUG] Sharded/EC volume: writes stall to ~0 and wedge the SPDK target on arm64/AIO, blocking volume deletion; plus CSI drops EC StorageClass params
- [BUG] v2 volume gets stuck in
Attachingin an IPv6 storage network environment - [BUG] CSI volume cloning doesn't respect node-/diskSelector
- [BUG] [v1.13.0-rc1] V2 volume DD easy to stuck in V2 interrupt mode
- [BUG] v1.12.1 default NetworkPolicies miss legitimate non-pod-labeled and host-originated callers, silently breaking CSI attach/detach
- [BUG] Replica rebuiliding may fail with
Input/output erroron v2 volumes - [BUG] Creating default disks with
createDefaultDiskLabeledNodessetting andcreate-default-disklabel doesn't work with BDF - [BUG] invalid block disks should not block Longhorn system uninstallation
- [BUG] The manifests in the /deploy directory of the longhorn/cli repository do not work due to an error in the command: field.
- [BUG] V2 replica rebuild destination failure faults the entire engine
- [BUG] v1 engine never starts: a stale
stoppedprocess record makes createInstance a silent no-op, volume stuck inattachingforever - [BUG] SPDK initialization fails on Talos OS with "nsenter: operation not permitted"
- [BUG] V2 expansion can report success while the engine remains at the old size
- [BUG] Recurring trim job fails with deadlock
- [BUG] Kernel Workqueue Lockup and Unstable RKE2 Service After Enabling LH V2 in Harvester
- [BUG] Failed to add v2 block disk with virtio-scsi BDF path
- [BUG] Longhorn 1.12.0: AWS chunked encoding not supported with OCI S3 buckets
- [BUG] Host OS nvmf-autoconnect connects kernel initiators to v2 replica subsystems, stalling volume attach/detach for minutes
- [BUG] Adding V2 disk using the "Default Data Path" /dev/disk/by-id/scsi-* path fails
- [BUG] backupvolume.longhorn.io not updated on cluster with nodes drained
- [BUG] Volume expansion snapshot CR name expand-<size> collides across volumes
- [BUG] System managed components don't respect System Managed Components Node Selector
- [BUG] Recurring-job pods report success after startup or volume execution errors
- [BUG] backupvolume.longhorn.io not created on cluster have nodes drained
- [BUG] Backup not started on cluster have nodes drained
- [BUG] v1 replica rebuild fails after a single gRPC TCP keepalive is lost
- [BUG] v1.12.1 chart: new default-on
networkPolicies.silently breaks Prometheus scraping of longhorn-manager (metrics/alerting blackout on patch upgrade)restrictInternalTraffic - [BUG] Longhorn UI is not running with PID 1
- [BUG][v1.6.2-rc2] Volume failed to recover after nodes reboot, pod failed to mount the volume with Input/output error
- [BUG] Snapshot CR is missing for backup CR of recurring job
- [BUG] V1 volumes not rebuilding after cluster shutdown
- [BUG] Incorrect Web Link in GUI
- [BUG] V2 backup/snapshot can leave NVMe/TCP frontend or dm device stale, causing pod EIO on attached volumes
- [BUG] Longhorn Helm Chart NetworkPolicies do not honor new RKE2 "rke2-traefik" ingress controller
- [BUG] v2 volume repeated replica reuse failure
- [BUG] Longhorn may try to attach volumes to a node without valid IM pod during the clone
- [BUG]
Backup Listing With More Than 1000 Backupsfails on v2 volume due to an empty replica address in the backup status - [BUG] Interrupt Mode V2 stuck high CPU after workload completed
- [BUG] Encrypted V2 volume size is 16MB short of the claimed size
- [BUG] Transient SPDK lvol metadata failure can permanently fault a healthy v2 replica
- [BUG] CSI pods do not respect anti-affinity preset update
- [BUG] V2 encrypted volume keeps switching between Attaching and Detaching state after expand operation
- [BUG] Fail to restore a v2 volume from a full backup if a previous backup is corrupted
- [BUG] V2 Data Engine: UBLK fails with EINVAL on Linux kernel 6.17.0
- [BUG] csi.*ReplicaCount Helm values silently no-op on existing csi-* deployments (only applied at first creation)
- [BUG] v2 volume may crash again after the auto reattachment
- [BUG] V2 Encrypted Volume Restore Fails
- [BUG] (chart) ArgoCD OutOfSync when using Gateway API
- [BUG] volume expansion stuck
- [BUG] expanding the volume fails
- [BUG] System Backup RecurringJob retention prunes newest CR — sorts by Status.CreatedAt (zero for Error/racing CRs)
- [BUG] CSI components may have 0 running replica during upgrade
- [BUG] Misconfigured and deleted Talos node causes Driver Deployer to constantly restart
- [BUG] V2 instance-manager liveness probe errors ('test: -eq: unary operator expected') -> self-kill -> node-wide replica fault under rebuild load
- [BUG] V2 Instance Manager panics in backupstore when S3 volume.cfg is missing, taking all node volumes offline
- [BUG] V2 backup readers race snapshot cleanup and panic the instance manager
- [BUG] Recurring snapshot-cleanup aborts the entire run when any volume's replica is rebuilding
- [BUG] Longhorn block disk remains Ready=False and Schedulable=False when NVMe device is bound to vfio-pci
- - [BUG] v2 volume stuck
attachingwith Storage Network enabled because theEngineFrontendtarget uses the engine pod IP instead ofStorageIP[13351](https://github.com/longhorn/longhorn/issues/13351) - @yangchiu @c3y1huang - [BUG] Migration Engine Can Be Unexpectedly Deleted If the Target Node Is Still in Readiness Transition
additions (28)
- Longhorn v1.13.0 supports live upgrade of V2 volumes, so the V2 instance managers can be upgraded without detaching the volumes, preventing disruption during Longhorn upgrades.
- Starting with v1.13.0, V2 linked-clone volumes also support restore operations.
- Introduced in v1.12.1, storage sharding is an experimental V2 Data Engine feature.
- the
data-engine-iobuf-large-pool-sizeanddata-engine-iobuf-small-pool-sizesettings control the size of the SPDK buffer pools used by the V2 Data Engine. - Longhorn v1.13.0 makes it easier to protect related volumes by letting you snapshot a group of volumes with a single request from the Longhorn UI, kubectl, or Kubernetes
VolumeGroupSnapshotobjects through CSI. - Longhorn v1.13.0 adds an
age-basedretention policy for snapshot, backup, and system backup recurring jobs - Longhorn v1.13.0 adds the
volumeTopologyStorageClass parameter (any,zonal, orregional) - Longhorn v1.13.0 improves pod scheduling by allowing kube-scheduler to check actual Longhorn disk capacity and, when possible, place a restarted pod on the node that already holds all of its replicas.
- Since v1.12.1, Longhorn creates ingress
NetworkPolicyresources for its internal components by default. - -
networkPolicies.andv1DataEngineInitiatorSourceCIDRs networkPolicies.allow traffic that the v1.12.1 policies blocked.recoveryBackendAdditionalIngressPorts - -
networkPolicies.lets Prometheus and other scrapers reach longhorn-manager metrics on TCP port 9500.metricsScrapeSources - Longhorn v1.13.0 runs the CSI controller sidecars under a dedicated
longhorn-csi-service-accountinstead of the sharedlonghorn-service-account. - You can turn this off with the
csi.Helm value after removingallowControllerSecretAccess csi.parameters from Longhorn StorageClasses.storage. k8s. io/provisioner-secret-* - [FEATURE] Allow configuring Gateway API filters
- [IMPROVEMENT] backup target availability status metric
- [IMPROVEMENT] Add longhorn-cli command to check if the user's environment supports IOMMU
- [IMPROVEMENT] Make
ignoreSigningHeadersconfigurable - [IMPROVEMENT][Helm] Allow setting spec.trafficDistribution
- [IMPROVEMENT] Add metrics to collect information about V2 data engine usage
- [IMPROVEMENT] Add a pod filter to the Workload/Pod detail popup
- [IMPROVEMENT] Expose topologySpreadConstraints for longhorn-ui in the Helm chart
- [IMPROVEMENT] v2 volumes: allow configuring the NVMe-TCP initiator I/O queue count (nr-io-queues)
- [IMPROVEMENT] Add usage metrics for volume size
- [IMPROVEMENT] Add metrics to collect information about LONGHORN_DISTRO
- [IMPROVEMENT] Exporting a volume from a single replica via crictl command
- [IMPROVEMENT] Chart value to allow metrics scrapers through the longhorn-manager NetworkPolicy
- [IMPROVEMENT] Help large replicas on full disks reclaim the space and squash all data into the heads
- - [IMPROVEMENT] Add opt-in PodDisruptionBudget for the Longhorn UI Deployment [13466](https://github.com/longhorn/longhorn/issues/13466) - @yangchiu @somaz94
value changes (9)
- interrupt mode for the V2 Data Engine is now fully event-driven. SPDK reactors wait for I/O events instead of polling continuously, with only low-frequency background checks remaining, significantly reducing CPU usage for idle or low I/O workloads.
- [IMPROVEMENT] v2 data engine: gate the replica host-ACL restriction on cluster-wide instance manager version
- [IMPROVEMENT] CSI ignores EC StorageClass parameters (silent replicated fallback)
- [IMPROVEMENT] (chart) Precedence of image.*.*.registry
- [IMPROVEMENT] Truncate the attached-to pod list in the volume table for volumes with many pods
- [IMPROVEMENT] Steer host RPS away from SPDK reactor cores
- [IMPROVEMENT] updateBackupCompressionMethod may write the Volume even when the method is unchanged
- [IMPROVEMENT] Improving error transparency for volume attachment failure
- [IMPROVEMENT] V2 volume write I/O stalls(~10s) when a replica is removed during migration
constraints (2)
- [IMPROVEMENT] Support mTLS encrypted communication for remaining gRPC services in instance manager
- - [FEATURE] Support Kubernetes CPU Manager for Longhorn V2 instance-manager SPDK CPU assignment [13248](https://github.com/longhorn/longhorn/issues/13248) - @yangchiu @mantissahz
defaults (1)
- Longhorn v1.13.0 sets
data-engine-cpu-isolation-enabledto{"v2":"true"}by default for new installations
A weekly email arrives when a release needs action. Like the security patches and breaking changes in this release.