# hami v2.7.0

2025-09-26 · [Original release notes](https://github.com/Project-HAMi/HAMi/releases/tag/v2.7.0) · [Full summary](https://ratatosk.io/en/releases/hami/v2.7.0)

Security 3 · Breaking changes 14 · Important fixes 14 · Enhancements 15 · Dependency updates 9 · Other changes 37

## Security (3)

### HAMi scheduler Helm RBAC now uses scoped roles

- Affects: HAMi Helm chart installations using the `hami-scheduler` service account and its generated RBAC bindings

Previously, the chart gave `hami-scheduler` the broad `cluster-admin` role. The chart now grants explicit permissions for HAMi's required resources and namespace-level lease operations, and binds the scheduler to the built-in `system:kube-scheduler` and `system:volume-scheduler` roles.

> * feat(scheduler-role): use a scoped-down role for scheduler by @Antvirf in https://github.com/Project-HAMi/HAMi/pull/1152

### HAMi image package updates for Trivy-reported issues

- Affects: Deployments using the HAMi image built from this Dockerfile (Inferred)

Before this release, Trivy reported security issues in the image, but the PR does not identify them individually. The build now runs `dnf update -y` on the `nvidia/cuda:12.6.3-base-ubi8` stage while excluding `cuda-toolkit-config-common` and `cuda-toolkit-12-config-common`; the PR says this helps resolve the findings, not that each is confirmed fixed.

> * fix: deal with security issues reported by Trivy in image by @Shouren in https://github.com/Project-HAMi/HAMi/pull/1189

### Go build configuration security fix

- Affects: HAMi deployments built from its image or source (Inferred)
- [CVE-2025-47907](https://www.cve.org/CVERecord?id=CVE-2025-47907)

The build image and `go.mod` previously used Go 1.24.4. The PR does not identify the affected code path or runtime conditions.

> * fix(security): resolve issues reported by Code scanning in Security by @Shouren in https://github.com/Project-HAMi/HAMi/pull/1280

## Breaking changes (14)

### vGPU allocation metrics gain new metrics and deprecation labels

- Affects: monitoring and automation consuming HAMi's vGPU allocation metrics (Inferred)
- Removed in: v2.8.0

Monitoring now receives `vGPUMemoryAllocated` and `vGPUCoreAllocated`; the existing `vGPUPodsDeviceAllocated`, `vGPUMemoryPercentage`, and `vGPUCorePercentage` metrics continue with a `deprecated_version="v2.8.0"` label and help text announcing their planned deprecation.

Evidence: “This pr removes and replaces it with two metrics that already exist.” https://github.com/Project-HAMi/HAMi/pull/1169

> * remove redundant metrics for vgpu allocation by @FouoF in https://github.com/Project-HAMi/HAMi/pull/1169

### HAMi now defaults GPU core limits to 100

- Affects: NVIDIA GPU workloads requesting the GPU-count resource with exclusive memory and no explicit GPU-core resource in requests or limits

Previously, HAMi left GPU cores unset in this case, with `nvidia.defaultCores` remaining 0 by default. Newly admitted matching pods receive a core-resource limit of 100, and `GPUDeviceCoreAllocated` and `vGPUCoreAllocated` reflect 100; existing pod specs change only when recreated or reapplied, while explicit core values and non-exclusive-memory cases remain unchanged.

Evidence: “The GPU core resource (config.ResourceCoreName, default nvidia.com/gpucores) must be absent from both limits and requests; explicit values are never overwritten.” https://github.com/Project-HAMi/HAMi/pull/1354

> * feat(nvidia): default gpucores=100 when memory is exclusive and cores… by @xrwang8 in https://github.com/Project-HAMi/HAMi/pull/1354

### HAMi webhook can replace Kubernetes's default scheduler name

- Affects: HAMi webhook-admitted pods with `Pod.Spec.SchedulerName` set to `DefaultSchedulerName`, and installations configuring scheduler settings through the chart or flag

The behavior is controlled by the chart setting `forceOverwriteDefaultScheduler` or the scheduler flag `--force-overwrite-default-scheduler`; both default to true.

Evidence: “+ rootCmd.Flags().BoolVar(&config.ForceOverwriteDefaultScheduler, "force-overwrite-default-scheduler", true, "Overwrite schedulerName in Pod Spec when set to the const DefaultSchedulerName in https://k8s.io/api/core/v1 package")” https://github.com/Project-HAMi/HAMi/pull/1163

> * fix: Add option for overwrite schedulerName by @Shouren in https://github.com/Project-HAMi/HAMi/pull/1163

### GPU device numeric conversions now use checked casts

- Affects: node annotations with device counts, memory, or core values; NVIDIA MIG health events; and device CPU affinity (Inferred)

Direct casts could produce incorrect converted values, and node-annotation numeric fields were cast to `int32` without a range check. Now, an unrepresentable annotation value returns a decode error, an unconvertible CPU-affinity value returns nil, and unconvertible or nonmatching MIG health-event IDs are skipped.

Evidence: “affinity, err := safecast.ToUint(node)” https://github.com/Project-HAMi/HAMi/pull/1183

> * fix: using go-safecast to fix incorrect conversion of numbers by @Shouren in https://github.com/Project-HAMi/HAMi/pull/1183

### Metax sGPU QoS policy selection via Pod annotation

- Affects: Metax sGPU jobs

Previously, Metax sGPU admission neither defaulted an omitted policy nor validated a supplied one. Now, omitted annotations assign `best-effort`; `best-effort`, `fixed-share`, and `burst-share` are accepted, while other values are rejected at admission.

Evidence: “Support Metax sGPU Qos Policy: user can specify qos policy for job” https://github.com/Project-HAMi/HAMi/pull/1123

> * Support Metax sGPU Qos Policy by @Kyrie336 in https://github.com/Project-HAMi/HAMi/pull/1123

### Metax sGPU fitting rejects unhealthy devices

- Affects: Metax sGPU device scheduling (Inferred)

A device with `Health=false` fails the fit check and cannot be selected for the request.

> * Support Metax sGPU device health check by @Kyrie336 in https://github.com/Project-HAMi/HAMi/pull/1295

### Helm chart version and default image tag updated to 2.7.0

- Affects: Helm chart installations using the default image tag, and systems that read chart version metadata (Inferred)

The chart’s `appVersion` changes from `2.6.1` to `2.7.0`.

> * update chart version and hami-core by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/1369

### NVIDIA device scheduling now honors `ResourceQuota` limits

- Affects: NVIDIA device requests in namespaces with applicable `ResourceQuota` limits (Inferred)

Applicable device-resource limits are used during scheduling. The scheduler also exposes `QuotaUsed` metrics labeled with the quota namespace, name, and limit.

Evidence: “This pr add resource quota support for Nvidia multi-device request and whole device request.” https://github.com/Project-HAMi/HAMi/pull/1359

> * NVIDIA Resourcequota by @FouoF in https://github.com/Project-HAMi/HAMi/pull/1359

### String processing now uses `strings.SplitSeq`

- Affects: NVIDIA, Enflame, Hygon, or Kunlun device support and utility string processing (Inferred)

These loops previously created a slice with `strings.Split` before iterating over its elements. They now range over `strings.SplitSeq` instead, which the change describes as more efficient.

Evidence: “The `SplitSeq` method is introduced in strings package since Go 1.24.0, and ranging over `SplitSeq` is more efficient” https://github.com/Project-HAMi/HAMi/pull/1239

> * refactor: Ranging over SplitSeq is more efficient by @Shouren in https://github.com/Project-HAMi/HAMi/pull/1239

### vGPUmonitor switches to an informer-backed local pod cache

- Affects: setups running vGPUmonitor

Previously, vGPUmonitor frequently queried the API server to check whether pods existed and reconcile container information; it now updates a local cache from informer events and reconciles periodically. Container-list refresh is 30 seconds by default, while informer resync is five minutes and configurable with `HAMI_RESYNC_INTERVAL`.

Evidence: “This PR adds an informer-based pod cache mechanism to the vGPUmonitor component to significantly reduce Kubernetes API server load.” https://github.com/Project-HAMi/HAMi/pull/1250

> * feat: add informer-based pod cache to reduce API server load by @miaobyte in https://github.com/Project-HAMi/HAMi/pull/1250

### `Devices.Fit` annotation parameter removed

- Affects: Go integrations that implement or call HAMi's `device.Devices` interface, including custom device implementations (Inferred)

Previously, annotation-dependent checks used an annotation map passed separately from the pod. They now use annotations from the supplied pod.

Evidence: “- Fit(devices []*DeviceUsage, request ContainerDeviceRequest, annos map[string]string, pod *corev1.Pod, nodeInfo *NodeInfo, allocated *PodDevices) (bool, map[string]ContainerDevices, string) + Fit(devices []*DeviceUsage, request ContainerDeviceRequest, pod *corev1.Pod, nodeInfo *NodeInfo, allocated *PodDevices) (bool, map[string]ContainerDevices, string)” https://github.com/Project-HAMi/HAMi/pull/1343

> * refactor: Remove annotation in Devices interfaces by @Shouren in https://github.com/Project-HAMi/HAMi/pull/1343

### Helm chart image values moved to nested per-image objects

- Affects: Helm chart image overrides using the old scalar fields or separate image settings (Inferred)

Previously, image names were scalar values, with settings such as pull policy and tag in separate fields; those settings now belong in per-image objects.

> * add global image for chart by @calvin0327 in https://github.com/Project-HAMi/HAMi/pull/1133

### Node locking switches to `Patch` as the chart removes Node `update` permission

- Affects: HAMi deployments using `pkg/util/nodelock` and the chart-managed scheduler role

Previously, node locking used `Update` on Nodes, and the chart granted the scheduler `get`, `list`, `patch`, `update`, and `watch` on Nodes.

Evidence: “- verbs: ["get", "list", "patch", "update", "watch"] + verbs: ["get", "list", "patch", "watch"]” https://github.com/Project-HAMi/HAMi/pull/1192

> * perf(util/nodelock): Use clientset Patch instead of Update. by @mayooot in https://github.com/Project-HAMi/HAMi/pull/1192

### Ascend 910A and 910B3 resource names corrected

- Affects: Ascend 910A or 910B3 device configurations and resource names (Inferred)

The 910B entry previously used `huawei.com/Ascend910B`; it now uses `huawei.com/Ascend910B3`, and the 910A and 910B3 devices are identified separately. Existing configurations must use the corrected names.

Evidence: “This pr changed device resource name, so the user must update their configs.” https://github.com/Project-HAMi/HAMi/pull/1315

> * cleanup: Clear and correct ascend device name by @FouoF in https://github.com/Project-HAMi/HAMi/pull/1315

## Important fixes (14)

### Kunlun node scores revised and device overrides added to defaults

- Affects: node scoring for Kunlun devices and registered device types (Inferred)

When scoring nodes with Kunlun devices, some cases returned lower values (for example, 2000 for a difference of -1) based on the change between current and previous `countbubble` values. For registered device types, an override replaced the default score only when positive; scores now add the device-specific override to the default.

> * fix: override node socre failure for kunlun #1137 by @ouyangluwei163 in https://github.com/Project-HAMi/HAMi/pull/1138

### Multi-node scheduler override scoring accuracy fixed

- Affects: multi-node scheduling with user node-policy override scoring (Inferred)

Override scoring now uses a captured device-usage snapshot for each node's scoring calculation.

> * fix: Multi-node scoring nodes are inaccurate by @ouyangluwei163 in https://github.com/Project-HAMi/HAMi/pull/1147

### Admission webhook logs now use Pod identity

- Affects: HAMi admission webhook logs (Inferred)

When logging admission requests, `req.UID` could differ from the Pod UID, and `req.Name` could be empty on CREATE if the client omitted the name. Logs now use the Pod's namespace, name, and UID, including in denial, admission, and error-related entries.

> * fix: wrong Pod's UID and emtpy Pod's name in log of webhook.go by @Shouren in https://github.com/Project-HAMi/HAMi/pull/1092

### Incorrect NVIDIA NUMA node results fixed

- Affects: NVIDIA GPU topology parsing with HAMi's NVIDIA device plugin (Inferred)

When parsing NVIDIA NUMA topology, the parser could return the wrong NUMA node; the multi-GPU test expects index 4 to resolve to `1`.

> * fix: fix err which retrieved incorrect NUMA node information issue #1275 by @abstractmj in https://github.com/Project-HAMi/HAMi/pull/1276

### Admission webhook preserves explicit Pod scheduler choices

- Affects: Pods with a nonempty `spec.schedulerName` that differs from HAMi's configured scheduler, or with no configured scheduler name

Previously, the webhook could override an explicitly assigned scheduler while continuing to process the Pod. It now admits matching Pods without further webhook handling.

> * fix: Skip admission webhook when Pod's scheduler is already assigned. by @ghostloda in https://github.com/Project-HAMi/HAMi/pull/1041

### Ascend 910B4-1 device support added

- Affects: Setups using Ascend 910B4-1 devices (Inferred)

The chart configuration includes the device’s resource identifiers and the `vir05_1c_8g` and `vir10_3c_16g` templates.

> * FIx CI, add 910B4-1 template and fix vGPUmonitor metrics error by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/1345

### NVIDIA vGPUmonitor metric calculations corrected

- Affects: setups using NVIDIA vGPUmonitor metrics (Inferred)

Previously, the calculations iterated over every slot in `s.sr.procs`, rather than only the first `procnum` slots. They now use only the process slots counted by `procnum`.

> * FIx CI, add 910B4-1 template and fix vGPUmonitor metrics error by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/1345

### NVIDIA MIG mode-switch startup error fixed

- Affects: NVIDIA device-plugin setups with devices supported by configured MIG templates

When `operatingmode` changed from `mig` to `hami-core` and `nvidia-mig-parted` did not turn off MIG, `hami-core` ran with an error. The non-`mig` startup path now creates and applies a MIG-disabled configuration for supported devices.

> * featue: mig mode-change #1116 by @ouyangluwei163 in https://github.com/Project-HAMi/HAMi/pull/1124

### Iluvatar device fitting now processes devices in forward order

- Affects: Iluvatar device scheduling and allocation (Inferred)

Creating an Iluvatar pod could error when fitting traversed the device list in reverse.

> * fix: An error occurred while create Iluvatar pod by @ouyangluwei163 in https://github.com/Project-HAMi/HAMi/pull/1149

### Concurrent map-access panic in scheduler scoring fixed

- Affects: Concurrent node scoring by the HAMi scheduler (Inferred)

When one goroutine wrote to a map while another read or wrote it during `calcScore`, concurrent map access could panic. The fitting-node and failed-node result writes now use separate mutexes, preventing that panic in this code path.

> * fix: concurrent map writes error in scheduler.calcScore #1269 by @Shouren in https://github.com/Project-HAMi/HAMi/pull/1270

### HAMi NodeLock cleanup for Bind errors and dangling locks

- Affects: HAMi scheduler NodeLock for Pod binding and device allocation

If the owner Pod was deleted after the scheduler set its NodeLock but before device allocation finished, subsequent Pods needing the same node could remain pending until the lock expired; a `PatchPodAnnotations` error during Bind also left the lock uncleared. This release clears the lock on that Bind error and allows dangling locks to be removed before expiry, including by non-holder Pods under timeout or dangling conditions.

> * fix: release dangling node lock by @peachest in https://github.com/Project-HAMi/HAMi/pull/1271

### Scheduler device allocation now includes containers without device requests

- Affects: Multi-container pods with device requests (Inferred)

Previously, allocation omitted a container with no device request when a later container requested a device. In the tested two-container case, the allocation now includes an empty entry for the first container and the device allocation for the second.

> * Fix: device allocation missing containers with no device request by @FouoF in https://github.com/Project-HAMi/HAMi/pull/1299

### NVIDIA MIG allocation now matches equal-memory templates

- Affects: NVIDIA MIG device scheduling

Before this fix, a request whose memory exactly matched a MIG template could fail to match a device and return `CardNotFoundCustomFilterRule`.

> * If the mem applied for the Mig device is the same as the template value,>will result in CardNotFoundCustom Filter Rule. by @zgqqiang in https://github.com/Project-HAMi/HAMi/pull/1179

### Unhealthy NVIDIA GPU scheduling flaw fixed

- Affects: Pod scheduling that requests NVIDIA GPUs

When a Pod requested an NVIDIA GPU with `health=false`, the scheduler still admitted it, but the Pod did not run. The fit check now rejects unhealthy GPUs before binding and logs card details at verbosity 5.

> * bugfix: Nvidia card abnormal pod will still continue to schedule by @zgqqiang in https://github.com/Project-HAMi/HAMi/pull/1336

## Enhancements (15)

- Topology-aware scheduling for 100%-core Metax sGPU allocations (To use: Enable `metaxsGPUTopologyAware` or set `metax-tech.com/sgpu-topology-aware` to `true`.)
- Kunlunxin scheduling supports four-card requests across NUMA boundaries
- Kunlunxin vXPU resource support
- Whole-card scheduling for Enflame GCU S60 (To use: `gcu-k8s-device-plugin >= 2.0` on nodes)
- AWS Neuron device and device-core allocation added
- Aggregated scheduling-failure details in Pod events
- Configurable HAMi scheduler node-lock timeout (To use: Optional `--node-lock-timeout` flag; defaults to 5 minutes.)
- Optional disabling of the HAMi admission webhook (To use: Helm value `scheduler.admissionWebhook.enabled` set to `false` (defaults to `true`))
- Scheduler adds an unknown-node allocation log
- HAMi node-lock timeout is configurable (To use: Set `HAMI_NODELOCK_EXPIRE` to a parseable duration, or set `scheduler.nodeLockExpire` in Helm.)
- Helm chart device-plugin resource toggle (To use: `devicePlugin.enabled` value, which defaults to `true`)
- Kunlun resource-name mapping added to the chart and device initialization
- Kunlunxin device support and topology-aware scheduling (To use: `devices.kunlun.enabled: true` and `kunlunxin.com/xpu` configured as a custom resource)
- Global Helm chart registry and image-pull-secret settings (To use: Configure `global.imageRegistry` and/or `global.imagePullSecrets` to use the corresponding setting.)
- Topology-aware scheduling for NVIDIA GPUs (To use: Set `scheduler.defaultSchedulerPolicy.gpuSchedulerPolicy` to `topology-aware` when installing HAMi; set `ENABLE_TOPOLOGY_SCORE` to `true` in the `hami-device-plugin` DaemonSet; add `gpu-scheduler-policy=topology-aware` when starting `hami-scheduler`; and set `hami.io/gpu-scheduler-policy: topology-aware` in Pod annotations.)

## Dependency updates (9)

### `tensorflow/tensorflow` upgraded from `latest-gpu` to `2.20.0rc0-gpu`

> * [Snyk] Security upgrade tensorflow/tensorflow from latest-gpu to 2.20.0rc0-gpu by @wawa0210 in https://github.com/Project-HAMi/HAMi/pull/1243

8 more, none mentioning security

- `* fix: update int8Slice to uint8Slice for better type clarity and consistency by @yxxhero in https://github.com/Project-HAMi/HAMi/pull/1357`
- `* fix: update int8Slice to uint8Slice for better type clarity and consistency by @yxxhero in https://github.com/Project-HAMi/HAMi/pull/1357`
- `* fix: update int8Slice to uint8Slice for better type clarity and consistency by @yxxhero in https://github.com/Project-HAMi/HAMi/pull/1357`
- `* fix: update int8Slice to uint8Slice for better type clarity and consistency by @yxxhero in https://github.com/Project-HAMi/HAMi/pull/1357`
- `* build(deps): upgrade golang to 1.24.4 by @Shouren in https://github.com/Project-HAMi/HAMi/pull/1172`
- `* build(deps): Upgrade controller-runtime to 0.21.0 by @Shouren in https://github.com/Project-HAMi/HAMi/pull/1171`
- `* build(deps): Dump github.com/NVIDIA/nvidia-container-toolkit  by @Shouren in https://github.com/Project-HAMi/HAMi/pull/1170`
- `* [Misc] update hami-core version by @chaunceyjiang in https://github.com/Project-HAMi/HAMi/pull/1201`

## Other changes (37)

- Unified fit logic replaced with device-specific fitting
- Release-note grouping recognizes `kind/feature`, `kind/design`, and `kind/enhancement`
- `watchAndFeedback` moved to `feedback.go`
- Node-lock retries switch to client-go backoff
- Topology-aware scheduling for Ascend910 devices
- NVML access coordinated during NVIDIA MIG partitioning
- golint CI checks fixed
- End-to-end CI fixed
- golangci-lint error fixed
- NVIDIA PCI bus ID conversion now uses `uint8Slice`
- Known issue documented for MIG partitioning on mixed HAMi and NVIDIA-native GPU nodes
- Ascend 910B documentation points to the `ascend-device-plugin` project
- Moore Threads GPU-sharing deployment documentation updated
- `README` device entries now use labeled text links
- Ascend 910B support documentation link updated
- Repository prerelease version files updated for v2.6
- Shouren and ouyangluwei163 added as reviewers
- Node configuration information added to the project documentation
- CI Go image upgraded to 1.24.4
- Fit Function unit tests added for Enflame, Hygon, Metax, Mthreads, and Nvidia
- NVIDIA configuration now merges per-node defaults with overrides
- Unit tests added for Cambricon's Fit Function
- Unit tests added for Ascend's Fit Function
- Pod resource-request generation reuses generated requests
- Node-annotation patch failure log wording corrected
- DRI section text combined for readability
- Issue translation robot added
- Issue-translation robot added
- Bundled `libvgpu` revision updated
- Bundled `libvgpu` revision updated
- GitHub workflow action for closing stale issues and pull requests
- README for chart values added
- Device and scheduler logic moved into corresponding packages
- HAMi CI build-blocking issue fixed
- HAMi Helm chart adds a default for `scheduler.service.httpTargetPort`
- Kunlunxin documentation updated
- HAMi core/libvgpu revision updated

---

Summaries are produced by AI, so they can be wrong, and they are provided without warranty.
