RATATOSKRATATOSK
Sign in

hami

v2.11.0AI & ML
Nov 4, 2026

ACTION 6CHECK 7PLAN 2OTHER 176

A broad maintenance release that expands accelerator, scheduling, monitoring, remote-GPU, and deployment capabilities while correcting validation and performance issues. Operators should review changed defaults, the Kubernetes 1.23 requirement, stricter validation, dependency updates, and deprecated metrics before upgrading.

Action needed (6)

  • breakingHAMI-core mounting removed

    This release no longer mounts HAMI-core to the whole GPU allocation. Metrics are exported directly by NVML.

  • breakingGPU count, memory, and core defaults

    The default GPU count, memory, and cores are now set by default.

  • breakingGPU core percentage range validation

    GPU core percentage requests outside the range [0,100] are rejected.

  • breakingScheduler-owned allocation annotation validation

    Pods created with scheduler-owned allocation annotations are denied.

  • breakingAllocation size validation

    The device plugin rejects allocations larger than the container requests.

  • breakingRequesting annotations removed

    Requesting annotations are removed.

Check if affected (7)

  • breakingCaller authentication for /refit

    Applies if you use /refit.

    Caller authentication is now required on /refit.

  • breakingKubernetes 1.23 requirement

    Applies if you do not depend on Kubernetes 1.23.

    This release requires Kubernetes 1.23.

  • breakingdevice-config defaults from values.yaml

    Applies if you configure device-config.

    Only values in values.yaml are used as defaults for device-config.

  • breakingNamespace ResourceQuota checking in Fit

    Applies if you configure namespace ResourceQuota.

    Fit now checks the namespace ResourceQuota.

  • breakingHygon default core value

    Applies if you run hygon.

    The default core value for hygon is updated.

  • breakingInvalid device count rejection

    Applies if you run biren or cambricon.

    Invalid device counts of zero or negative values are rejected across biren and cambricon.

  • breakingNVIDIA fractional GPU memory validation

    Applies if you run nvidia.

    Fractional GPU memory percentages are rejected for NVIDIA devices.

Plan ahead (2)

  • deprecatedhami_container_device_memory_bytes metric deprecation

    Applies if you use hami_container_device_memory_bytes.

    The hami_container_device_memory_bytes metric is deprecated.

  • deprecatedlimit label on hami_resource_quota_used

    Applies if you use hami_resource_quota_used.

    The limit label of the hami_resource_quota_used metric is deprecated.

All 176 other recorded changesfixes 98 · additions 42 · value changes 23 · constraints 9 · renames 3 · removals 1

fixes (98)

  • Fix/scheduler metrics audit
  • fixing the reject unknown fields in device-config.yml
  • fix(nvidia): honor enableGetPreferredAllocation in GetDevicePluginOptions
  • fix(ascend): guard against empty ResourceMemoryName injection in admission
  • fix(ascend): do not overwrite ASCEND_VISIBLE_DEVICES for Ascend containers
  • fix(chart): use lists for device plugin extraEnvs defaults
  • fix(device-plugin): honor disable-core-limit flag
  • fix(monitor): replace MustNewConstMetric with safe helper
  • [kind/bug] Restore model-aware scheduling for Cambricon MLU devices
  • validate device-scoring-weights at admission
  • validate device resource names and defaults at startup
  • fix: disable MIG when using hami-core
  • guard allocation annotations on update, not just create
  • auto driverroot make mount unchange
  • fail closed on invalid HAMi device requests instead of binding GPU-less pods
  • select Lupine server using all pod container requests
  • surface collection failures
  • run library-copy init first
  • exclude NotReady Lupine servers
  • mount GPU limits at /overrideEnv for login sessions
  • stop mounting the host root for auto driver root
  • requeue pods waiting for a lupine fleet change
  • resolve dynamic MIG identities
  • watch profiling fields
  • Fix vgpu-monitor can't discover mig-instance properly
  • select quota-feasible GPU combinations in topology scheduling
  • serialize expired-lock recovery
  • don't cache node annotation as up-to-date when the patch fails
  • fix(device): reject malformed and negative device allocation annotations
  • fix(nvidia): collapse duplicate MIG candidates in topology scheduling
  • fix(skill): align vGPU metrics summary with actual codebase definitions
  • fix(scheduler): include init container requests in device locking
  • fix(dashboard): align node GPU memory ratio scale
  • fix(vGPUmonitor): gracefully skip GPU utilization metrics when unsupported
  • fix: NUMA refit follow-ups from post-merge review
  • fix(scheduler): keep the configured policy for unrecognized policy an…
  • fix(scheduler): account terminating pods replayed by informer sync
  • fix(scheduler): keep candidate nodes for non HAMi pods on the simulation path
  • fix(metax): let multi container pods spread over exclusive GPUs
  • fix(plugin): pair per device limits by UUID when aligning kubelet IDs
  • fix(awsneuron): cap whole-device core accounting to addressable cores
  • fix(enflame): keep DRS profile core costs within the device budget
  • fix(kunlun): make topology selection independent of device slice order
  • fix(amd): keep allocation annotations decodable for empty registered types
  • fix(device-plugin): remove panic(0) traps from getAPIDevices and dead util code
  • fix(monitor): reuse the node-scoped pod informer for metrics
  • fix(monitor): report metrics server listen failures
  • fix(chart): target the scheduler metrics port by name
  • fix(ascend): gate 910C module pair allocation on SuperPod
  • fix(metrics): drop the constant container index label
  • fix(scheduler): keep bare reasons in FilteringFailed events
  • fix(device-plugin): propagate IMEX channels into the NVIDIA plugin
  • fix(scheduler): use phase-aware accounting during NUMA refit
  • fix(scheduler): validate NUMA refit quota with container positions
  • fix(dashboard): use native node labels for host panels
  • fix(util): return the canonical GPU scheduler policy for padded annotations
  • fix(kunlun): compare elements instead of indices in calcscore sort
  • fix(scheduler): keep multi-slot device usage across annotation round trip
  • fix(monitor): count task priorities in a map, not a slice index
  • fix(chart): render service labels at the right indent level
  • fix(scheduler): clean annotationless pod deletes from cache
  • Fix Ascend webhook reject pod with 'vnpu-core' resource
  • fix(nvidia): guard Allocate against empty DevicesIds requests
  • fix(chart): honor kube-scheduler image pull secrets
  • fix(chart): keep scheduler livenessProbe inside the kubeScheduler container guard
  • fix(scheduler): prune printedLog when a node is deleted
  • fix(chart): keep workload labels on scraped HAMi metrics
  • fix(nvidia): make managed admission env mutation idempotent
  • fix(scheduler): serve the extender verbs on loopback, not the cluster port
  • fix(scheduler): verify pod identity before the extender acts on a request
  • fix(awsneuron): allocate by device index, not slice position
  • fix(iluvatar): read the scaled core limit as a per card value
  • fixing the awsneuron: derive core geometry per node
  • fix(scheduler): score devices against the request on model-typed backends
  • fix(monitor): drop duplicate NVIDIA prefix from device_type label
  • fix(device): enforce minimum limit across multiple ResourceQuota objects
  • fix(remotegpu): keep multi-container pod allocations on one Lupine server
  • fix(scheduler): report undecodable live Pod allocations
  • fix(amd): round core requests up to whole WGPs
  • fix(vGPUmonitor): make the MIG lock wait respect context cancellation
  • fix(scheduler): gpumem score before gpumem-percentage
  • fix(nodelock): keep an expired lock while its holder waits to be allocated
  • fix: defer Ascend memory trimming until node filtering
  • fix(util): serve only the pod the node lock names
  • fix(hygon): prevent dcucores integer overflow
  • fix(amd): judge node health by allocatable GPUs
  • fix(amd): read a registration without a type as the AMD type
  • fix(scheduler): keep HAMI_NODELOCK_EXPIRE unless --node-lock-timeout is set
  • fix(scheduler): restore steady-state usage on pod replay
  • Fix healthcheck always return true for some device vendors
  • guard the PSP and pin images by digest
  • fix(device-plugin): mark all replicas/MIG slices unhealthy on XID
  • fix(device-plugin): check MIG allocations against the registered profiles
  • fix(vGPUmonitor): skip terminated pods in per-container metrics
  • fix(scheduler): ignore negative percentage
  • fix(scheduler): store quota memory limits scaled by memoryFactor
  • fix(scheduler): hold allocLock when removing a pod's allocation
  • fix(chart): add actual NVML names to MIG allowlist

additions (42)

  • PodGroup Support
  • New schedule policy: mutex
  • Add template to vNPU-mode
  • add profiling support with dedicated HTTP bind address
  • expose scheduler kube-qps, kube-burst and kube-timeout as values
  • Flexible-MIG
  • Real-usage monitoring for Flexible-mig workloads
  • Allows running vllm on remote cuda
  • Schedule policy combinations
  • GPU/CPU Numa Align
  • track InitContainers
  • feat: add scheduler NUMA refit for device allocations
  • Add Prometheus metrics for scheduler leader and cache sync state
  • Feat/scheduler poddisruptionbudget
  • feat: add host PID broker server and device plugin integration
  • Expose Hardware Health and Error Metrics in vGPUmonitor
  • add auto detecting nvidiaDriverRoot
  • expose node-lock-retry-timeout and extender httpTimeout
  • allow a pod to prefer a MIG profile over 3g via annotation
  • pod level and container level overwrite env
  • add remote-gpu device backend for lupine-served GPUs
  • configurable per-card memory units
  • Add CDI support for dynamic MIG devices
  • schedule ENPU workloads with policy isolation
  • support amd.com/numa-bind for multi-GPU requests
  • add health probes
  • match nodeconfig by label selector
  • expose Ascend hami-core mode slot capacity
  • set a configurable runtimeClassName on AMD GPU Pods
  • add limitDelivery switch
  • honor device cordon
  • enhancement remote mode
  • introduce deviceSplitCount for both hami-core and enpu mode
  • add a redacted HAMi diagnostic bundle collector
  • select Lupine server pools by label
  • Add 'nvidia-smi' to remote client
  • add MIG monitoring support
  • add securityContext for the lupine server
  • introduce global.imageSource to switch between upstream and CN mirrors
  • support session image pull secrets in configuration and templates
  • add an unknown GPU process count metric
  • add admission.k8s.io/v1 to webhook admissionReviewVersions

value changes (23)

  • libvnpu performance optimization
  • Add Missing Node Label to Memory Controller Metric in vGPUmonitor
  • Optimize the Dynamic MIG instance lifecycle
  • fix(deps): update Go dependencies and bump golang.org/x/net to v0.60.0
  • chore: upgrade to go 1.27
  • chore: build the images with golang 1.27.2
  • score MIG placements and add backtracking fallback
  • stop deep-copying whole Pods into the usage snapshot
  • honor plugin-advertised hami-core percentage budget
  • index device UUIDs once per scrape
  • run GPUs for lupine instead of advertising them locally
  • build the usage snapshot only for candidate nodes
  • report memory and core resources to node capacity
  • honor the topology-aware policy by XGMI links and NUMA
  • whole-GPU hook bypass and direct NVML monitoring
  • enhance fields validator
  • honor preferred nodeAffinity in node score
  • bump Go to 1.27.3
  • pin the dcgm base image
  • bump DCGM to 4.7.0-2
  • go-dcgm to v1.4702.0
  • bump golang.org/x/net from 0.61.0 to 0.62.0
  • Update HAMi-core and image tag

constraints (9)

  • Support Cluster Autoscaler
  • Full AMD Support
  • Ascend 950 Support
  • account native sidecar container GPU resources
  • add OpenShift &SELinux helm chart support
  • whole-card/slice mixing guard
  • support all four NeuronCores on Inferentia1 devices
  • support gpumem-percentage memory requests
  • Configure Ascend950DT 9582 whole-card scheduling

renames (3)

  • rename DCU(HYGON) to HCU across resources, charts, and docs
  • move all vendor configs to devices.VENDOR
  • stable usage.cache name

removals (1)

  • fix(chart): drop unused pods and nodes verbs from the device-plugin ClusterRole
Add hami to your stack

A weekly email arrives when a release needs action. Like the breaking changes in this release.

Add to stack →
Official release note ↗All hami releases →