hami

v2.10.0AI & ML

2026年8月21日

運用時の不具合や挙動を幅広く修正し、スケジューリングやデバイス対応、設定、可観測性に関する変更を含む保守リリースです。不要になった機能の削除に加え、依存関係や実行環境、ツールチェーンの更新も含まれるため、アップグレード時は互換性と設定内容の確認が必要です。

要対応 (3)

securitytensorflow/tensorflowの更新
tensorflow/tensorflowを2.20.0rc0-gpuから2.21.0rc0-gpuへ更新しました。
securitytensorflow/tensorflowの更新
tensorflow/tensorflowを2.21.0rc0-gpuから2.21.0rc1-gpuへ更新しました。
securitygolangのセキュリティ更新
セキュリティ上の問題に対応するため、golangを更新しました。

影響確認 (7)

breakingHAMiメインチャートからのDRAコンポーネント削除
HAMiのメインチャートでDRAコンポーネントを利用している場合に該当します。
HAMiのメインチャートからDRAコンポーネントを削除しました。
remove dra from charts` by @archlitchi #2038
breakingWebhookによる特権コンテナの拒否
Webhookを利用している場合に該当します。
Webhookが特権コンテナを拒否するようになりました。
breaking非推奨スケジューラーポリシーConfigMapの削除
非推奨のスケジューラーポリシー用ConfigMapを設定している場合に該当します。
非推奨のscheduler policy configmapを削除しました。
breakingdevice pluginのホストネットワーク無効化
デバイスプラグインが実行される場合に該当します。
device pluginのホストネットワークを無効にしました。
breaking非推奨対象ラベルの削除
適用対象はリリースノートに明記されていません。
非推奨対象としてマークされたlabelsを削除しました。
breakingチャートからのDRA削除
チャートでDRAを利用している場合に該当します。
チャートからDRAを削除しました。
breakinghami-core-vNPUタスクのpostStartフック削除
hami-core-vNPUタスクを実行している場合に該当します。
hami-core-vNPUタスクからpostStartフックを削除しました。

その他の記録済み変更 195 件すべてsecurity fixes 1 · fixes 144 · additions 27 · value changes 12 · constraints 9 · removals 1 · renames 1

security fixes (1)

* security: add io.LimitReader to scheduler routes to prevent DoS

fixes (144)

- **Init Container Support** → - [feat(scheduler): correct resource accounting for init containers by @maishivamhoo123 #1773](https://github.com/Project-HAMi/HAMi/pull/1773) - [docs: design for init-container GPU resource accounting #2064](https://github.com/Project-HAMi/HAMi/pull/2064) - [docs: document ResourceQuota interaction for init container accounting #2535](https://github.com/Project-HAMi/HAMi/pull/2535)
- **HAMi-core mode for vnpu doesn't need to align device memory to template** → [Fix bug about auto-memory trim when using hami-core mode for vNPU by @archlitchi #2696](https://github.com/Project-HAMi/HAMi/pull/2696)
* fix(scheduler): bound-check MIG template/instance index parsed from UUID annotation by @mesutoezdil in https://github.com/Project-HAMi/HAMi/pull/2088
* fix(nvidia): detect NVLink through NVSwitch for topology scoring by @peachest in https://github.com/Project-HAMi/HAMi/pull/2103
* Fix: adopt to gangScheduling feature gates on k8s v1.36+ by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/2206
* fix: precedence bug in schedulerName check by @hoteye in https://github.com/Project-HAMi/HAMi/pull/1627
* fix: add nil checks to prevent panics in leaderelection by @haitwang-cloud in https://github.com/Project-HAMi/HAMi/pull/1603
* fix: panic on nil resourceReqs in scheduler calcScore by @yxxhero in https://github.com/Project-HAMi/HAMi/pull/1626
* Fix the issue Iluvatar device scheduling policy binpack and spread are reversed by @qiangwei1983 in https://github.com/Project-HAMi/HAMi/pull/1631
* fix: resolve cardinality explosion in Device_memory_desc_of_container by @maishivamhoo123 in https://github.com/Project-HAMi/HAMi/pull/1628
* fix: handle GetMemoryInfo ERROR_NOT_SUPPORTED for unified memory GPUs by @jsl9208 in https://github.com/Project-HAMi/HAMi/pull/1637
* Fix: optimize nodelock scalability with exponential backoff and listers by @maishivamhoo123 in https://github.com/Project-HAMi/HAMi/pull/1663
* fix apply kubescheduler config version by @CoderTH in https://github.com/Project-HAMi/HAMi/pull/1666
* fix: failing readiness probe when replica > 1 by @Shouren in https://github.com/Project-HAMi/HAMi/pull/1677
* fix(scheduler): correct slot usage prediction and add device type fil… by @maishivamhoo123 in https://github.com/Project-HAMi/HAMi/pull/1700
* fix vastai fit loop iteration direction by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1715
* fix: retain terminating pod in cache to prevent premature eviction by @maishivamhoo123 in https://github.com/Project-HAMi/HAMi/pull/1719
* fix(chart): derive ld.so.preload from devicePlugin.libPath to fix non-default path deployments by @ilia-medvedev in https://github.com/Project-HAMi/HAMi/pull/1714
* fix: support device allocation for multi-container with init containers by @haitwang-cloud in https://github.com/Project-HAMi/HAMi/pull/1650
* fix: Reponse with correct content-type by @Shouren in https://github.com/Project-HAMi/HAMi/pull/1604
* fix: suppress scheduler cleanup noise for unrelated vendors by @Yonsun-w in https://github.com/Project-HAMi/HAMi/pull/1749
* fix: global image tag always covers per-component image tag by @FouoF in https://github.com/Project-HAMi/HAMi/pull/1774
* fix(device-plugin): align kubelet allocation with scheduler annotations (#1741) by @xrwang8 in https://github.com/Project-HAMi/HAMi/pull/1743
* fix(chart): sanitize managedNodeSelector keys for environment variables by @almazkhalikov in https://github.com/Project-HAMi/HAMi/pull/1783
* fix: filter device does not work by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1817
* Fix: Handle Kernel 6.17 handshake edge cases in NVIDIA health checks by @maishivamhoo123 in https://github.com/Project-HAMi/HAMi/pull/1810
* fix: git errors when building with latest hami-core by @Shouren in https://github.com/Project-HAMi/HAMi/pull/1782
* fix: mig does not work by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1819
* fix(scheduler): guard against zero-value division in ComputeScore (#1… by @lin121291 in https://github.com/Project-HAMi/HAMi/pull/1820
* fix(ascend): check unmarshal error before iterating node devices by @mesutoezdil in https://github.com/Project-HAMi/HAMi/pull/1831
* fix(device): parse handshake annotation timestamp in local timezone by @mesutoezdil in https://github.com/Project-HAMi/HAMi/pull/1816
* fix: recover scheduling on nodes with stale Deleted_ handshake by @saiyam1814 in https://github.com/Project-HAMi/HAMi/pull/1843
* fix A100-80G template by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1847
* fix: allocation failed when using mig in CDI mode by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1826
* fix: propagate parse errors in DecodeNodeDevices by @mesutoezdil in https://github.com/Project-HAMi/HAMi/pull/1855
* fix: return error from DecodePodDevices on decode failure by @mesutoezdil in https://github.com/Project-HAMi/HAMi/pull/1857
* fix: propagate parse errors in DecodeContainerDevices by @mesutoezdil in https://github.com/Project-HAMi/HAMi/pull/1856
* Fix time-out logic incorrectly triggered by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/1879
* fix(quota): clamp Used to zero in RmUsage to prevent negative tracking by @mesutoezdil in https://github.com/Project-HAMi/HAMi/pull/1880
* fix(vGPUmonitor): prevent index out of bounds panic for priority above 1 by @mesutoezdil in https://github.com/Project-HAMi/HAMi/pull/1881
* fix(quota): roll back usage when Filter evicts a stale pod entry by @mesutoezdil
* fix: rollback quota usage when filter annotation patch fails by @bolin-dai
* fix(chart): add DRA resource RBAC for kube-scheduler on K8s 1.34+ by @spencercjh
* fix(scheduler): clean up podManager on Bind failures to prevent resource leak by @peachest
* fix(charts): change scheduler anti-affinity to preferred for rolling … by @lvkaili
* Fix: Skip nvidia-mig-parted when MIG is disabled by @maishivamhoo123
* fix_leaderNotify Block by @archlitchi
* fix(device): CheckUUID ignores noUseKey when useKey is also set by @mesutoezdil
* fix(device): prevent panic in CheckHealth when Requesting has no timestamp by @mesutoezdil
* fix(quota): clamp Used to zero in RmUsage to prevent negative tracking by @mesutoezdil
* fix: return snapshot in InspectAllNodesUsage to avoid data race by @DSFans2014
* fix(chart): declare metrics port on vGPU monitor container by @ntheanh201
* fix(nvidia): validate gpumem-percentage range to prevent silent unschedulability by @saiyam1814
* fix(webhook): count the resource to fit quota in webhook when reqnum > 1 by @DSFans2014
* fix(device-plugin): surface NVML error when MIG device placement cannot be determined by @saiyam1814
* fix(vgpu-init): correct source and destination path handling by @haitwang-cloud
* fix(nvidia): refresh cache on register annotation changes by @LynPtl
* fix(ascend): reject -core on hard-split vNPU and fix "nor"->"not" typo by @Wangmin362
* fix(ascend): stop whole-card/memory-less hami-core requests from bypassing the node mode gate by @Wangmin362
* fix(scheduler): accept a vastai-only or biren-only device configuration by @Wangmin362
* fix(device-plugin): avoid nil pointer panic in CheckHealth when checkHealth returns nil by @Wangmin362
* fix(scheduler): treat an empty use/nouse gpuuuid annotation as no constraint by @Wangmin362
* fix(cambricon): avoid scheduler panic when a leading container requests no MLU by @Wangmin362
* fix: skip owner check for legacy lock format by @DSFans2014
* fix: return deep copies from ListPodsInfo to prevent data race by @mesutoezdil
* fix(scheduler): avoid the racy read of the node cache in register() by @Wangmin362
* fix(scheduler): do not exit on a transient error updating the leader label by @Wangmin362
* fix(scheduler): scope per-type device list in fitInDevices by @imantaba
* fix: bound int32 conversion in enflame device parsing by @mesutoezdil
* fix(metrics): resolve high-cardinality leak in memory_allocated_bytes by @Aryanbhargava18
* fix(mthreads): guard against zero device count in GenerateResourceRequests by @imantaba
* fix(enflame): bound profile values before int32 conversion in Fit by @mesutoezdil
* fix(ascend): update ascend config according to latest driver by @FouoF
* fix(metrics): prevent invalid memory ratios for unknown total memory by @blackdragoon26
* fix(scheduler): handle ResourceQuota tombstones by @blackdragoon26
* fix(monitor): fix use-after-unmap crash in loadCache + raise cudevshr.go coverage to 90% by @adity1raut
* fix(device): guard nil maps in mthreads MutateAdmission by @imantaba
* fix(scheduler): preserve concurrent node locks by @ShiroKSH
* fix(device): correct AllocatedCardsInsufficientRequest count during partial allocation by @princexpoddar
* fix(vGPUmonitor): scan all devices in CheckBlocking by @imantaba
* fix(nvidia): handle zero-score topology combinations by @Eshiv-Pandey
* fix(nodelock): let a pod re-acquire its own still-valid node lock by @adity1raut
* fix(device): use HasPrefix and TrimPrefix in DelQuota by @SK8-infi
* fix: release node lock on allocate response failure by @cotishq
* fix(ascend): skip unhealthy devices in Fit() by @Eshiv-Pandey
* fix(device-plugin): release node lock when bind-phase patch fails by @blackdragoon26
* fix(vGPUmonitor): bound v0 spec metric sums to active proc slots by @imantaba
* fix(mthreads): correct invalid memory slice value in error message by @Eshiv-Pandey
* fix(device): use len(devices) for GenReason in kunlun and awsneuron by @blackdragoon26
* fix(scheduler): return deep copy in nodeManager.GetNode to prevent concurrent map read/write by @AyushSrivastava1818
* fix(nvidia): treat gpumem-percentage of 0 as unset by @Wangmin362
* fix(scheduler): return early when predicate body is nil by @hemantrajputt
* fix(scheduler): enforce ResourceQuota for every device backend by @Lakshya77089
* fix: add missing metric descriptors to Describe() in vGPUmonitor by @devGPP23
* fix(scheduler): apply ResourceQuota updates without dropping the limit by @Lakshya77089
* fix(monitor): skip dirs without underscore in Update instead of panicking by @Nakshatra480
* fix(vGPUmonitor): skip uninitialized short UUIDs instead of exit by @SK8-infi
* fix(metrics): replace dangerous MustNewConstMetric with safe helper in scheduler by @devGPP23
* fix(monitor): clamp SetDeviceSmLimit and SetDeviceMemoryLimit to maxDevices by @Nakshatra480
* fix(device): don't panic on asymmetric GPU P2P link data in calculateGPUPairScore by @Brijesh-Thakkar
* fix(scheduler): prevent nil body panic bug in Predicate and Bind handlers by @devGPP23
* fix(vGPUmonitor): skip exited process slots in v1 spec metric aggregations by @Nakshatra480
* fix (ascend) - validate module pair count and prevent under allocation for ascend 910C by @princexpoddar
* fix(monitor): clamp DeviceNum to maxDevices to prevent scrape-path panic by @Nakshatra480
* fix(quota): honor an explicit zero limit in ResourceQuota by @veyron-kairo
* fix(device-plugin): return error on failed Send in ListAndWatch by @AyushSrivastava1818
* fix(device-plugin): skip devices that do not support event based healthchecking by @Wangmin362
* fix(device,plugin): stop MIG usage corruption and fd leak by @adity1raut
* fix(amd): empty nouse-gputype annotation should not exclude every device by @adity1raut
* fix(device): skip empty comma-separated members in shared CheckType by @adity1raut
* fix(device-plugin): resolve stale annotation in multi-container Allocate by @peachest
* fix: skip unhealthy devices in Fit() for all non-nvidia backends by @Eshiv-Pandey
* fix(metrics): normalize AMD core allocation ratio to percentage by @yxxhero
* fix(ascend): guard nil Requests in MutateAdmission by @keshav9926
* fix(util): handle nil pod gracefully in utility functions to prevent … by @Rickydama3
* fix(monitor): ignore unrelated files when loading cache by @hyy321
* fix(cambricon): enforce ResourceQuota for percentage/whole-card memory requests by @adity1raut
* fix(monitor): use IsValidUUID to skip uninitialised device UUIDs in scrape path by @Nakshatra480
* fix(scheduler): copy node info into nodeManager instead of storing the caller's by @Lakshya77089
* fix(monitor): add v0.MinSize and fix file-size discriminator for v0 cache by @Nakshatra480
* fix(device): hand out copies from PodManager instead of stored pointers by @Lakshya77089
* fix(nvidia): clean up node health bookkeeping on deletion by @AyushSrivastava1818
* fix(cambricon): prevent node lock leaks by reading from apiserver in ReleaseNodeLock by @jianzhangbjz
* fix(hygon,metax): guard int32 narrowing in GenerateResourceRequests by @Eshiv-Pandey
* fix(awsneuron): handle init-container annotation indexes by @Eshiv-Pandey
* fix: Handle HTTP response write errors in scheduler routes by @shinigami-777
* Fix bug about auto-memory trim when using hami-core mode for vNPU by @archlitchi
* fix(scheduler): remove stale device vendors on zero-device update by @AyushSrivastava1818
* fix(metrics): add device_type label to node memory ratio metrics (#2370) by @princexpoddar
* fix(vGPUmonitor): add missing node label to host GPU metrics by @SK8-infi
* fix(ascend): guard int32 narrowing in GenerateResourceRequests by @Eshiv-Pandey
* fix(scheduler): detect stale pod device allocations during usage reconciliation by @AyushSrivastava1818
* fix(scheduler): rollback partial device locks by @AyushSrivastava1818
* fix(scheduler): allow multiple MIG instances on one GPU by @saiyam1814
* fix(awsneuron): prevent odd NeuronCore request truncation by @Eshiv-Pandey
* fix(scheduler): don't cache a nil NodeUsage when node is missing from snapshot by @adity1raut
* fix(scheduler): reconcile health on device discovery error by @AyushSrivastava1818
* fix(kunlun): honour use-gpuuuid and nouse-gpuuuid annotations by @Lakshya77089
* fix(device): reject int32-overflowing memory requests in iluvatar and mthreads by @veyron-kairo
* fix(scheduler): reject filter requests with no Pod instead of panicking
* fix: use atomic load/store for shared-region limit fields
* fix(device-plugin): make MigInstanceManager own its NVML init/shutdown
* fix(scheduler): preserve per-container slot count when collapsing pod device usage
* fix(scheduler): add missing return in ondelpod

additions (27)

- **New schedule policy (mutex)** → [feat: new mutex GPU scheduling policy by @mesutoezdil #2011](https://github.com/Project-HAMi/HAMi/pull/2011)
- **Numa align (CPU and GPU)** → [device-plugin: opt-in NUMA topology for vGPU replicas by @saiyam1814 #2065](https://github.com/Project-HAMi/HAMi/pull/2065)
- **Biren series** → - [feat: add support for biren device by @DSFans2014 #1711](https://github.com/Project-HAMi/HAMi/pull/1711) - [docs: mark Biren as completed in roadmap #1979](https://github.com/Project-HAMi/HAMi/pull/1979)
- **AMD Mi300x series** → - [feat(amd): support amd vgpu by @FouoF #2290](https://github.com/Project-HAMi/HAMi/pull/2290) - [design(amd): update amd design docs #2067](https://github.com/Project-HAMi/HAMi/pull/2067) - [docs: add AMD Instinct vGPU design draft (#1707) by @kenji-mido #1985](https://github.com/Project-HAMi/HAMi/pull/1985) - [docs: fix typo and mark AMD and MIG as supported in roadmap #1975](https://github.com/Project-HAMi/HAMi/pull/1975)
- **mock-device-plugin support new NPU Template** → [feat(ascend): support 910C vNPU templates vir05_1c_16g and vir10_3c_32g by @ouyangluwei163 #2005](https://github.com/Project-HAMi/HAMi/pull/2005)
* feat: add vGPUmonitor --metrics-bind-address flag by @dongjiang1989 in https://github.com/Project-HAMi/HAMi/pull/1613
* feat: add promtheus serviceMonitor in helm-charts by @dongjiang1989 in https://github.com/Project-HAMi/HAMi/pull/1614
* feat: add serviceMonitor for device plugin by @dongjiang1989 in https://github.com/Project-HAMi/HAMi/pull/1633
* feat: check resource quota in webhook by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1605
* feat: add support for vastai device by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1645
* feat(helm): add config namespaceSelector & objectSelector for webhook by @haitwang-cloud in https://github.com/Project-HAMi/HAMi/pull/1653
* feat: add Ascend ResourceCoreName to support hami-vnpu-core virtualization by @ashergaga in https://github.com/Project-HAMi/HAMi/pull/1771
* feat: add Ascendxxx-core resource by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1804
* feat: support node filtering based on hami-vnpu-core annotation by @ashergaga in https://github.com/Project-HAMi/HAMi/pull/1812
* feat: add enableGetPreferredAllocation flag by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1824
* feat(helm): make device plugin security context configurable via values by @ilia-medvedev in https://github.com/Project-HAMi/HAMi/pull/1887
* feat(scheduler): support template node simulation filtering by @spencercjh in https://github.com/Project-HAMi/HAMi/pull/2046
* feat(helm): add switch for admission webhook namespaceSelector by @spencercjh in https://github.com/Project-HAMi/HAMi/pull/2091
* feat(nvidia): support RTX PRO 6000 Blackwell dynamic MIG by @shkatara in https://github.com/Project-HAMi/HAMi/pull/2135
* feat(dashboards): add importable Grafana dashboard for HAMi GPU metrics by @imantaba in https://github.com/Project-HAMi/HAMi/pull/2301
* feat(scheduler): support per-pod device scoring weights by @blackdragoon26 in https://github.com/Project-HAMi/HAMi/pull/2469
* feat: add hami_host_gpu_memory_controller_utilization_ratio metric by @manoj-1407 in https://github.com/Project-HAMi/HAMi/pull/2616
* feat(scheduler): export hami_resource_quota_limit gauge metric by @SK8-infi in https://github.com/Project-HAMi/HAMi/pull/2715
* Support live per-GPU cordon via node annotation by @mohityadav8 in https://github.com/Project-HAMi/HAMi/pull/2298
* Helm - Render nvidia.overwriteEnv from values with a default of false.
* Configure ASCEND_VISIBLE_DEVICES env for container and RuntimeClassName for pods
* device-plugin: opt-in NUMA topology for vGPU replicas

value changes (12)

- **Change Compile image to ubi8 to support all GLIBC versions** → - [build(docker): align HAMi with HAMi-core compile image by @spencercjh #1958](https://github.com/Project-HAMi/HAMi/pull/1958) - [docs: drop stale glibc upper bound from prerequisites #2102](https://github.com/Project-HAMi/HAMi/pull/2102)
* fix: harden HTTP servers and migconfig file permissions by @mesutoezdil in https://github.com/Project-HAMi/HAMi/pull/2100
* Update HAMi-core and version to v2.10 by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/2759
* add device type label in metrics
* vGPUmonitor: skip devices with invalid UTF-8 UUID during container init
* update nvidia_dp and nvidia_container_runtime module
* Upgrade Go to 1.26.2
* bump HAMi-DRA version to v0.2.0
* build(deps): bump github.com/ccoveille/go-safecast/v2
* build: bump Go toolchain to 1.26.5
* chore: bump Go builder to 1.26.6
* chore: bump golang.org/x/mod to v0.40.0

constraints (9)

- **Support podgroup** → - [feat(scheduler): retry NodeLock in Bind for PodGroup members by @lin121291 #2066](https://github.com/Project-HAMi/HAMi/pull/2066) - [Fix: adopt to gangScheduling feature gates on k8s v1.36+ by @archlitchi #2206](https://github.com/Project-HAMi/HAMi/pull/2206)
- **Flexible Mig** → [refactor(MIG): support MIG instance dynamic allocation/deallocation by @FouoF #2378](https://github.com/Project-HAMi/HAMi/pull/2378)
- **Scheduler policy combination** → [feat: support comma-separated gpu-scheduler-policy combinations by @mesutoezdil #2621](https://github.com/Project-HAMi/HAMi/pull/2621)
- **vNPU Compat mode / Heterogeneous Ascend Mode (vNPU + HAMi-core)** → [Feat: Allow mode-agnostic Ascend pods to schedule across both vNPU-template and HAMi-core nodes by @ouyangluwei163 #2035](https://github.com/Project-HAMi/HAMi/pull/2035)
* feat: support module-pair allocation for Ascend 910C devices in SuperPod environments by @ashergaga in https://github.com/Project-HAMi/HAMi/pull/1610
* feat: Supports multi-device requests with hami-vnpu-core enabled. by @ashergaga in https://github.com/Project-HAMi/HAMi/pull/1837
* feat: Support enflame drs by @FouoF in https://github.com/Project-HAMi/HAMi/pull/1872
* feat(helm): allow passing device-config content directly via values.yaml by @shkatara in https://github.com/Project-HAMi/HAMi/pull/2277
* chore: using LeaderElectionConfiguration for kubernetes v1.23 and newer version

removals (1)

* Remove 'deleted' annotations when registering

renames (1)

* feat(metrics): align prometheus metric and label names with best practices by @MyoungHaSong in https://github.com/Project-HAMi/HAMi/pull/1644

hamiをスタックに追加

対応が必要なリリースが出たときに、週次メールでお知らせします。今回のセキュリティパッチと破壊的変更も、その一例です。

スタックに追加 →

公式リリースノート ↗hamiのリリース一覧 →