RATATOSKRATATOSK
Sign in

Volcano

v1.16.0Orchestration & Management
Oct 14, 2026

ACTION 2CHECK 9OTHER 89

A largely additive release expands multi-tenant queueing, topology-aware scheduling, observability, platform support, and Helm and CLI capabilities, with most new behavior opt-in. Operators should review the security updates and the Kubernetes and Go compatibility requirements before upgrading.

Action needed (2)

  • securityDependency updates for upstream security advisories

    Updates Prometheus, golang.org/x/crypto, x/net, x/text, OpenTelemetry, etcd, gRPC, and other indirect dependencies to address upstream security advisories.

  • breakingThe --csi-storage flag default

    The --csi-storage flag now defaults to true.

Check if affected (9)

  • securityDRA capacity aggregation fix, GHSA-j38h-7pfq-cxmw

    Applies if you run DRA capacity accounting.

    The DRA capacity aggregation path no longer permits tenant-controlled device or minimum-task counts to trigger excessive iteration. This addresses GHSA-j38h-7pfq-cxmw.

  • breakingCluster Queue spec.allowedNamespaces authorization

    Applies if you use NamespaceQueue.

    Cluster Queue authorization now uses spec.allowedNamespaces: omitted or empty lists deny NamespaceQueue attachment, while ["*"] allows all namespaces.

  • breakingNamespaceQueue authorization and readiness checks

    Applies if you use NamespaceQueue.

    NamespaceQueue creation now requires the namespace and an authorized parent Queue, namespace-local RBAC permissions for namespacequeues, and confirmation of Ready=True and Authorized=True before workloads are submitted.

  • breakingcustom.hypernode_controller_mode migration sequence

    Applies if you configure custom.hypernode_controller_mode.

    Integrated installations must disable custom.hypernode_controller_mode and wait for the controller-manager rollout before enabling standalone, so only one process reconciles HyperNodes.

  • breakingHyperNode topology plugin interfaces

    Applies if you use HyperNodeGradientForJobFn, HyperNodeGradientForSubJobFn, or HyperNodeCandidateFn.

    Custom topology plugins using these HyperNode extension points must adapt to the new behavior and rebuild: HyperNodeGradientForJobFn, HyperNodeGradientForSubJobFn, and HyperNodeCandidateFn.

  • breakingKubernetes 1.36.1 and Go 1.26 build requirements

    Applies if you do not use Go 1.26.

    The release builds against Kubernetes 1.36.1 and requires Go 1.26 for source builds.

  • breakingKubernetes 1.23 deployment support

    Applies if you use Kubernetes 1.23.

    Deployments on Kubernetes 1.23 are unsupported.

  • breakingHyperNode gradient return types

    Applies if you use HyperNode gradient return types.

    Custom topology plugins must be rebuilt for the new HyperNode gradient return types.

  • breakingTTL cleanup for Helm admission-init Jobs

    Applies if you use Helm admission-init Jobs.

    Helm admission-init Jobs now use TTL cleanup instead of immediate deletion after a successful hook.

All 89 other recorded changessecurity fixes 1 · additions 33 · fixes 32 · value changes 12 · constraints 8 · defaults 2 · removals 1

security fixes (1)

  • Adds release SBOM generation and Security Insights metadata

additions (33)

  • The UnschedulableJobCache records why a Job could not be scheduled and uses plugin-provided event hints to decide when it should be evaluated again.
  • The feature is disabled by default. Enable it with the scheduler feature gate: bash helm upgrade --install volcano volcano-sh/volcano \ --namespace volcano-system --create-namespace --version 1.16.0 \ --set custom.scheduler_feature_gates="UnschedulableJobCache=true"
  • NamespaceQueue introduces namespace-local queue management while preserving administrator control over the cluster-level resource hierarchy.
  • Controller-maintained authorization and readiness conditions, with final scheduler readiness checks before admission.
  • The new fairshare plugin tracks cumulative resource-seconds per namespace within a queue and applies exponential half-life decay.
  • Optional enqueue gating and ConfigMap persistence across scheduler restarts.
  • PodGroup anti-affinity expresses isolation at the workload-group level. It selects peer PodGroups by labels and namespace selectors and applies the separation rule at a named or numbered HyperNode tier.
  • The standalone controller provides an independent deployment and operational boundary while retaining the same HyperNode API, discovery configuration, and implementation.
  • A dedicated vc-hypernode-controller-manager binary, image, Deployment, RBAC, and Helm configuration.
  • Adds hami-vnpu-core integration for Ascend memory virtualization and compute time slicing.
  • Checks Ascend health using allocatable devices and adds vGPU UUID-based allowlist/denylist constraints.
  • Adds inqueue resource metrics, queue workload-state metrics, and volcano_eviction_transactions_total{action}.
  • Makes scheduler job/task updater concurrency configurable
  • logs loaded scheduler configuration for troubleshooting
  • Adds namespace columns and stable sorting to vcctl pod list
  • release automation for Linux/macOS amd64/arm64 CLI archives, checksums, and SLSA provenance
  • Adds scheduling.volcano.sh/v1beta1 NamespaceQueue
  • Adds spec.topologyAffinity.podGroupAntiAffinity
  • Adds Numatopology.spec.podAllocations
  • Adds Session.AddHintProvider and QueueingHint-style event registration for unschedulable-job retries
  • Adds the Queue annotation volcano.sh/nodegroup-resource-limits
  • Adds the fairshare.* arguments
  • Adds Job plugin option svc: ["--inject-hosts-env=false"]
  • Adds Pod annotations volcano.sh/vgpu-use-gpuuuid and volcano.sh/vgpu-nouse-gpuuuid
  • Adds --job-updater-worker-num and --task-updater-worker-num
  • Adds custom.hypernode_controller_mode
  • Adds custom.admission_init_ttl (default 3600 seconds)
  • retention annotations for the existing Volcano CRDs
  • Configures GOMAXPROCS using container CPU quotas across Volcano components.
  • Update CRDs and RBAC together with the control-plane components
  • Gang-aware gangPreempt and gangReclaim remain explicit actions
  • Caps how much a queue can allocate from each nodegroup while allowing placement in other permitted groups.
  • Adds pod-level resource settings and accounts for restartable init containers/native sidecars and Pod overhead in effective requests used by the Volcano agent.

fixes (32)

  • Registers upstream PreFilter hooks for CSI limits, node affinity, and volume-zone checks
  • avoiding phantom reservations when a later plugin rejects admission
  • Includes gate-reserved resources from descendant queues in ancestor quota checks
  • Preserves Completed state after finished Pods are deleted
  • Returns Closed queues to Closing on resync when PodGroups remain
  • Uses PodGroup MinResources for Volcano Jobs that do not yet have Pods
  • closes a remaining overflow path so large device counts cannot wrap into negative quota usage
  • preventing false NUMA policy failures
  • preventing Pods such as Spark executors with newly created PVCs from remaining permanently outside scheduler consideration
  • restores node state after unsuccessful updates
  • Adds missing nil checks for PodGroup-related cache paths
  • Honors NeedContinueAllocating after PrePredicate failures
  • Retains the allocation-rate policy default range when its arguments are omitted.
  • Formats effectively unlimited resource values consistently in scheduler resource strings.
  • Corrects UpdateHyperNode rebuild logic
  • Honors nominated HyperNodes without hard topology mode
  • Guards annotation patches by Pod UID when Pods are recreated
  • Avoids deleting the PodGroup for targeted Pod restarts
  • Falls back to replica count when partition minPartitions is omitted.
  • Returns status-update failures for retry
  • Synchronizes TaskInfo allocation annotations across binding and rollback
  • Prevents idle-memory underflow
  • Fixes large-cluster vGPU scheduling coordination.
  • Corrects allocation resource accounting
  • Rechecks device feasibility after tentative evictions before committing a reclaim decision.
  • Completes or requeues popped Pods on early exits
  • Rejects non-positive worker counts
  • Matches CNI plugins by type rather than display name.
  • Deduplicates Pod lists by namespace and name
  • reports all invalid subgroup and hierarchical-weight entries
  • preserves the existing Volcano CRDs and their custom resources on uninstall
  • avoiding scheduler stalls for very large counts

value changes (12)

  • Bounded retries**: A configurable watchdog retries cached Jobs after a maximum skip duration, defaulting to five minutes. The cache is in memory and is cleared on scheduler restart.
  • Historical usage ordering within a queue, with a default four-hour half-life.
  • Fairshare influences pending-job ordering and optional enqueue admission. It does not preempt or reclaim already-running jobs. Persistence is disabled by default.
  • The usage plugin now maintains a session-level shadow load cache for assigned Pods whose usage is not yet visible in metrics.
  • The metrics.interval setting defines the monitoring delay window and defaults to 30 seconds.
  • Reuses the session NodeMap, batches nodeorder PreScore/Score work, and executes active score plugins in one parallel node traversal before plugin-owned normalization and aggregation.
  • Updates Kubernetes dependencies to 1.36.1, the Go toolchain requirement to 1.26
  • Applies upstream DynamicResources Score and plugin-owned NormalizeScore to prioritized device requests
  • Uses the upstream VolumeBinding implementation in place of Volcano's fork
  • Updates golang.org/x/crypto to v0.56.0, x/net to v0.58.0, x/text to v0.41.0, and Prometheus to v0.311.3
  • Updates CNI, cgroups, filesystem notifications, Elasticsearch, Prometheus common, and the OTLP trace exporter
  • Updates e2e-framework, Ginkgo, Gomega, and gomonkey.

constraints (8)

  • Namespace-local parent/child hierarchies, with cluster/<name> references for cluster Queue parents.
  • Explicit workload references through spec.queue: namespace/<name>, resolved within the workload's namespace. Bare queue names continue to reference cluster Queues.
  • NamespaceQueue hierarchies allow five levels below a cluster Queue by default. Set --max-namespacequeue-depth on both the controller manager and admission webhook to change that limit.
  • Each term must specify exactly one of topologyTierName and topologyTier. Preferred terms additionally require a weight from 1 to 100. Required terms omit weight.
  • Enforcement applies to allocate. BestEffort Pods placed through backfill do not receive this isolation guarantee.
  • Kubernetes 1.36 support
  • Fairshare must precede DRF for historical-usage ordering to take effect.
  • update resource-exporter to a version supporting spec.podAllocations

defaults (2)

  • controller-manager, standalone, and disabled deployment modes, with integrated mode retained as the default.
  • NamespaceQueue and UnschedulableJobCache are disabled by default.

removals (1)

  • The final implementation no longer injects a PostStart hook.
Add Volcano to your stack

A weekly email arrives when a release needs action. Like the security patches and breaking changes in this release.

Add to stack →
Official release note ↗All Volcano releases →