hami
v2.10.0AI & MLA substantial operator-focused maintenance release with broad bug and behavior fixes, alongside new scheduling, device, configuration, and observability capabilities. It also updates security-relevant dependencies and the runtime/toolchain while removing obsolete functionality that may affect compatibility and configuration.
Action needed (3)
security
tensorflow/tensorflowupgraded to2.21. 0rc0-gpu The
tensorflow/tensorflowdependency is upgraded from2.to20. 0rc0-gpu 2..21. 0rc0-gpu security
tensorflow/tensorflowupgraded to2.21. 0rc1-gpu The
tensorflow/tensorflowdependency is upgraded from2.to21. 0rc0-gpu 2..21. 0rc1-gpu security
golangsecurity upgradeThe
golangruntime is upgraded to address a security issue.
Check if affected (7)
breakingDRA components removed from the HAMi main chart
Applies if you use DRA components in the HAMi main chart.
The HAMi main chart no longer includes
DRAcomponents.breakingWebhook denial of privileged containers
Applies if you use the webhook.
The admission webhook now denies privileged containers.
breakingDeprecated scheduler policy configmap removed
Applies if you configure the deprecated scheduler policy configmap.
The deprecated scheduler policy configmap is removed.
breakingDevice plugin host networking disabled
Applies if the device plugin runs.
Host networking is disabled for the device plugin.
breakingDeprecated labels removed
Applicability is not stated in the release notes.
Labels marked for deprecation are removed.
breaking
DRAremoved from chartsApplies if you use DRA in charts.
DRAis removed from the charts.breaking
postStarthook removed fromhami-core-vNPUtasksApplies if you run hami-core-vNPU tasks.
The
postStarthook is removed forhami-core-vNPUtasks.
All 195 other recorded changessecurity fixes 1 · fixes 144 · additions 27 · value changes 12 · constraints 9 · removals 1 · renames 1
security fixes (1)
- * security: add io.LimitReader to scheduler routes to prevent DoS
fixes (144)
- - **Init Container Support** → - [
feat(scheduler): correct resource accounting for init containersby @maishivamhoo123 #1773](https://github.com/Project-HAMi/HAMi/pull/1773) - [docs: design for init-container GPU resource accounting#2064](https://github.com/Project-HAMi/HAMi/pull/2064) - [docs: document ResourceQuota interaction for init container accounting#2535](https://github.com/Project-HAMi/HAMi/pull/2535) - - **HAMi-core mode for vnpu doesn't need to align device memory to template** → [
Fix bug about auto-memory trim when using hami-core mode for vNPUby @archlitchi #2696](https://github.com/Project-HAMi/HAMi/pull/2696) - * fix(scheduler): bound-check MIG template/instance index parsed from UUID annotation by @mesutoezdil in https://github.com/Project-HAMi/HAMi/pull/2088
- * fix(nvidia): detect NVLink through NVSwitch for topology scoring by @peachest in https://github.com/Project-HAMi/HAMi/pull/2103
- * Fix: adopt to gangScheduling feature gates on k8s v1.36+ by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/2206
- * fix: precedence bug in schedulerName check by @hoteye in https://github.com/Project-HAMi/HAMi/pull/1627
- * fix: add nil checks to prevent panics in leaderelection by @haitwang-cloud in https://github.com/Project-HAMi/HAMi/pull/1603
- * fix: panic on nil resourceReqs in scheduler calcScore by @yxxhero in https://github.com/Project-HAMi/HAMi/pull/1626
- * Fix the issue Iluvatar device scheduling policy binpack and spread are reversed by @qiangwei1983 in https://github.com/Project-HAMi/HAMi/pull/1631
- * fix: resolve cardinality explosion in Device_memory_desc_of_container by @maishivamhoo123 in https://github.com/Project-HAMi/HAMi/pull/1628
- * fix: handle GetMemoryInfo ERROR_NOT_SUPPORTED for unified memory GPUs by @jsl9208 in https://github.com/Project-HAMi/HAMi/pull/1637
- * Fix: optimize nodelock scalability with exponential backoff and listers by @maishivamhoo123 in https://github.com/Project-HAMi/HAMi/pull/1663
- * fix apply kubescheduler config version by @CoderTH in https://github.com/Project-HAMi/HAMi/pull/1666
- * fix: failing readiness probe when replica > 1 by @Shouren in https://github.com/Project-HAMi/HAMi/pull/1677
- * fix(scheduler): correct slot usage prediction and add device type fil… by @maishivamhoo123 in https://github.com/Project-HAMi/HAMi/pull/1700
- * fix vastai fit loop iteration direction by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1715
- * fix: retain terminating pod in cache to prevent premature eviction by @maishivamhoo123 in https://github.com/Project-HAMi/HAMi/pull/1719
- * fix(chart): derive ld.so.preload from devicePlugin.libPath to fix non-default path deployments by @ilia-medvedev in https://github.com/Project-HAMi/HAMi/pull/1714
- * fix: support device allocation for multi-container with init containers by @haitwang-cloud in https://github.com/Project-HAMi/HAMi/pull/1650
- * fix: Reponse with correct content-type by @Shouren in https://github.com/Project-HAMi/HAMi/pull/1604
- * fix: suppress scheduler cleanup noise for unrelated vendors by @Yonsun-w in https://github.com/Project-HAMi/HAMi/pull/1749
- * fix: global image tag always covers per-component image tag by @FouoF in https://github.com/Project-HAMi/HAMi/pull/1774
- * fix(device-plugin): align kubelet allocation with scheduler annotations (#1741) by @xrwang8 in https://github.com/Project-HAMi/HAMi/pull/1743
- * fix(chart): sanitize managedNodeSelector keys for environment variables by @almazkhalikov in https://github.com/Project-HAMi/HAMi/pull/1783
- * fix: filter device does not work by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1817
- * Fix: Handle Kernel 6.17 handshake edge cases in NVIDIA health checks by @maishivamhoo123 in https://github.com/Project-HAMi/HAMi/pull/1810
- * fix: git errors when building with latest hami-core by @Shouren in https://github.com/Project-HAMi/HAMi/pull/1782
- * fix: mig does not work by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1819
- * fix(scheduler): guard against zero-value division in ComputeScore (#1… by @lin121291 in https://github.com/Project-HAMi/HAMi/pull/1820
- * fix(ascend): check unmarshal error before iterating node devices by @mesutoezdil in https://github.com/Project-HAMi/HAMi/pull/1831
- * fix(device): parse handshake annotation timestamp in local timezone by @mesutoezdil in https://github.com/Project-HAMi/HAMi/pull/1816
- * fix: recover scheduling on nodes with stale Deleted_ handshake by @saiyam1814 in https://github.com/Project-HAMi/HAMi/pull/1843
- * fix A100-80G template by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1847
- * fix: allocation failed when using mig in CDI mode by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1826
- * fix: propagate parse errors in DecodeNodeDevices by @mesutoezdil in https://github.com/Project-HAMi/HAMi/pull/1855
- * fix: return error from DecodePodDevices on decode failure by @mesutoezdil in https://github.com/Project-HAMi/HAMi/pull/1857
- * fix: propagate parse errors in DecodeContainerDevices by @mesutoezdil in https://github.com/Project-HAMi/HAMi/pull/1856
- * Fix time-out logic incorrectly triggered by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/1879
- * fix(quota): clamp Used to zero in RmUsage to prevent negative tracking by @mesutoezdil in https://github.com/Project-HAMi/HAMi/pull/1880
- * fix(vGPUmonitor): prevent index out of bounds panic for priority above 1 by @mesutoezdil in https://github.com/Project-HAMi/HAMi/pull/1881
- * fix(quota): roll back usage when Filter evicts a stale pod entry by @mesutoezdil
- * fix: rollback quota usage when filter annotation patch fails by @bolin-dai
- * fix(chart): add DRA resource RBAC for kube-scheduler on K8s 1.34+ by @spencercjh
- * fix(scheduler): clean up podManager on Bind failures to prevent resource leak by @peachest
- * fix(charts): change scheduler anti-affinity to preferred for rolling … by @lvkaili
- * Fix: Skip nvidia-mig-parted when MIG is disabled by @maishivamhoo123
- * fix_leaderNotify Block by @archlitchi
- * fix(device): CheckUUID ignores noUseKey when useKey is also set by @mesutoezdil
- * fix(device): prevent panic in CheckHealth when Requesting has no timestamp by @mesutoezdil
- * fix(quota): clamp Used to zero in RmUsage to prevent negative tracking by @mesutoezdil
- * fix: return snapshot in
InspectAllNodesUsageto avoid data race by @DSFans2014 - * fix(chart): declare metrics port on vGPU monitor container by @ntheanh201
- * fix(nvidia): validate gpumem-percentage range to prevent silent unschedulability by @saiyam1814
- * fix(webhook): count the resource to fit quota in webhook when reqnum > 1 by @DSFans2014
- * fix(device-plugin): surface NVML error when MIG device placement cannot be determined by @saiyam1814
- * fix(vgpu-init): correct source and destination path handling by @haitwang-cloud
- * fix(nvidia): refresh cache on register annotation changes by @LynPtl
- * fix(ascend): reject -core on hard-split vNPU and fix "nor"->"not" typo by @Wangmin362
- * fix(ascend): stop whole-card/memory-less hami-core requests from bypassing the node mode gate by @Wangmin362
- * fix(scheduler): accept a vastai-only or biren-only device configuration by @Wangmin362
- * fix(device-plugin): avoid nil pointer panic in CheckHealth when checkHealth returns nil by @Wangmin362
- * fix(scheduler): treat an empty use/nouse gpuuuid annotation as no constraint by @Wangmin362
- * fix(cambricon): avoid scheduler panic when a leading container requests no MLU by @Wangmin362
- * fix: skip owner check for legacy lock format by @DSFans2014
- * fix: return deep copies from ListPodsInfo to prevent data race by @mesutoezdil
- * fix(scheduler): avoid the racy read of the node cache in register() by @Wangmin362
- * fix(scheduler): do not exit on a transient error updating the leader label by @Wangmin362
- * fix(scheduler): scope per-type device list in fitInDevices by @imantaba
- * fix: bound int32 conversion in enflame device parsing by @mesutoezdil
- * fix(metrics): resolve high-cardinality leak in memory_allocated_bytes by @Aryanbhargava18
- * fix(mthreads): guard against zero device count in GenerateResourceRequests by @imantaba
- * fix(enflame): bound profile values before int32 conversion in Fit by @mesutoezdil
- * fix(ascend): update ascend config according to latest driver by @FouoF
- * fix(metrics): prevent invalid memory ratios for unknown total memory by @blackdragoon26
- * fix(scheduler): handle ResourceQuota tombstones by @blackdragoon26
- * fix(monitor): fix use-after-unmap crash in loadCache + raise cudevshr.go coverage to 90% by @adity1raut
- * fix(device): guard nil maps in mthreads MutateAdmission by @imantaba
- * fix(scheduler): preserve concurrent node locks by @ShiroKSH
- * fix(device): correct AllocatedCardsInsufficientRequest count during partial allocation by @princexpoddar
- * fix(vGPUmonitor): scan all devices in CheckBlocking by @imantaba
- * fix(nvidia): handle zero-score topology combinations by @Eshiv-Pandey
- * fix(nodelock): let a pod re-acquire its own still-valid node lock by @adity1raut
- * fix(device): use HasPrefix and TrimPrefix in DelQuota by @SK8-infi
- * fix: release node lock on allocate response failure by @cotishq
- * fix(ascend): skip unhealthy devices in Fit() by @Eshiv-Pandey
- * fix(device-plugin): release node lock when bind-phase patch fails by @blackdragoon26
- * fix(vGPUmonitor): bound v0 spec metric sums to active proc slots by @imantaba
- * fix(mthreads): correct invalid memory slice value in error message by @Eshiv-Pandey
- * fix(device): use len(devices) for GenReason in kunlun and awsneuron by @blackdragoon26
- * fix(scheduler): return deep copy in nodeManager.GetNode to prevent concurrent map read/write by @AyushSrivastava1818
- * fix(nvidia): treat gpumem-percentage of 0 as unset by @Wangmin362
- * fix(scheduler): return early when predicate body is nil by @hemantrajputt
- * fix(scheduler): enforce ResourceQuota for every device backend by @Lakshya77089
- * fix: add missing metric descriptors to Describe() in vGPUmonitor by @devGPP23
- * fix(scheduler): apply ResourceQuota updates without dropping the limit by @Lakshya77089
- * fix(monitor): skip dirs without underscore in Update instead of panicking by @Nakshatra480
- * fix(vGPUmonitor): skip uninitialized short UUIDs instead of exit by @SK8-infi
- * fix(metrics): replace dangerous MustNewConstMetric with safe helper in scheduler by @devGPP23
- * fix(monitor): clamp SetDeviceSmLimit and SetDeviceMemoryLimit to maxDevices by @Nakshatra480
- * fix(device): don't panic on asymmetric GPU P2P link data in calculateGPUPairScore by @Brijesh-Thakkar
- * fix(scheduler): prevent nil body panic bug in Predicate and Bind handlers by @devGPP23
- * fix(vGPUmonitor): skip exited process slots in v1 spec metric aggregations by @Nakshatra480
- * fix (ascend) - validate module pair count and prevent under allocation for ascend 910C by @princexpoddar
- * fix(monitor): clamp DeviceNum to maxDevices to prevent scrape-path panic by @Nakshatra480
- * fix(quota): honor an explicit zero limit in ResourceQuota by @veyron-kairo
- * fix(device-plugin): return error on failed Send in ListAndWatch by @AyushSrivastava1818
- * fix(device-plugin): skip devices that do not support event based healthchecking by @Wangmin362
- * fix(device,plugin): stop MIG usage corruption and fd leak by @adity1raut
- * fix(amd): empty nouse-gputype annotation should not exclude every device by @adity1raut
- * fix(device): skip empty comma-separated members in shared CheckType by @adity1raut
- * fix(device-plugin): resolve stale annotation in multi-container Allocate by @peachest
- * fix: skip unhealthy devices in Fit() for all non-nvidia backends by @Eshiv-Pandey
- * fix(metrics): normalize AMD core allocation ratio to percentage by @yxxhero
- * fix(ascend): guard nil Requests in MutateAdmission by @keshav9926
- * fix(util): handle nil pod gracefully in utility functions to prevent … by @Rickydama3
- * fix(monitor): ignore unrelated files when loading cache by @hyy321
- * fix(cambricon): enforce ResourceQuota for percentage/whole-card memory requests by @adity1raut
- * fix(monitor): use IsValidUUID to skip uninitialised device UUIDs in scrape path by @Nakshatra480
- * fix(scheduler): copy node info into nodeManager instead of storing the caller's by @Lakshya77089
- * fix(monitor): add v0.MinSize and fix file-size discriminator for v0 cache by @Nakshatra480
- * fix(device): hand out copies from PodManager instead of stored pointers by @Lakshya77089
- * fix(nvidia): clean up node health bookkeeping on deletion by @AyushSrivastava1818
- * fix(cambricon): prevent node lock leaks by reading from apiserver in ReleaseNodeLock by @jianzhangbjz
- * fix(hygon,metax): guard int32 narrowing in GenerateResourceRequests by @Eshiv-Pandey
- * fix(awsneuron): handle init-container annotation indexes by @Eshiv-Pandey
- * fix: Handle HTTP response write errors in scheduler routes by @shinigami-777
- * Fix bug about auto-memory trim when using hami-core mode for vNPU by @archlitchi
- * fix(scheduler): remove stale device vendors on zero-device update by @AyushSrivastava1818
- * fix(metrics): add device_type label to node memory ratio metrics (#2370) by @princexpoddar
- * fix(vGPUmonitor): add missing node label to host GPU metrics by @SK8-infi
- * fix(ascend): guard int32 narrowing in GenerateResourceRequests by @Eshiv-Pandey
- * fix(scheduler): detect stale pod device allocations during usage reconciliation by @AyushSrivastava1818
- * fix(scheduler): rollback partial device locks by @AyushSrivastava1818
- * fix(scheduler): allow multiple MIG instances on one GPU by @saiyam1814
- * fix(awsneuron): prevent odd NeuronCore request truncation by @Eshiv-Pandey
- * fix(scheduler): don't cache a nil NodeUsage when node is missing from snapshot by @adity1raut
- * fix(scheduler): reconcile health on device discovery error by @AyushSrivastava1818
- * fix(kunlun): honour use-gpuuuid and nouse-gpuuuid annotations by @Lakshya77089
- * fix(device): reject int32-overflowing memory requests in iluvatar and mthreads by @veyron-kairo
- * fix(scheduler): reject filter requests with no Pod instead of panicking
- * fix: use atomic load/store for shared-region limit fields
- * fix(device-plugin): make MigInstanceManager own its NVML init/shutdown
- * fix(scheduler): preserve per-container slot count when collapsing pod device usage
- * fix(scheduler): add missing return in ondelpod
additions (27)
- - **New schedule policy (mutex)** → [
feat: new mutex GPU scheduling policyby @mesutoezdil #2011](https://github.com/Project-HAMi/HAMi/pull/2011) - - **Numa align (CPU and GPU)** → [
device-plugin: opt-in NUMA topology for vGPU replicasby @saiyam1814 #2065](https://github.com/Project-HAMi/HAMi/pull/2065) - - **Biren series** → - [
feat: add support for biren deviceby @DSFans2014 #1711](https://github.com/Project-HAMi/HAMi/pull/1711) - [docs: mark Biren as completed in roadmap#1979](https://github.com/Project-HAMi/HAMi/pull/1979) - - **AMD Mi300x series** → - [
feat(amd): support amd vgpuby @FouoF #2290](https://github.com/Project-HAMi/HAMi/pull/2290) - [design(amd): update amd design docs#2067](https://github.com/Project-HAMi/HAMi/pull/2067) - [docs: add AMD Instinct vGPU design draft (#1707)by @kenji-mido #1985](https://github.com/Project-HAMi/HAMi/pull/1985) - [docs: fix typo and mark AMD and MIG as supported in roadmap#1975](https://github.com/Project-HAMi/HAMi/pull/1975) - - **mock-device-plugin support new NPU Template** → [
feat(ascend): support 910C vNPU templates vir05_1c_16g and vir10_3c_32gby @ouyangluwei163 #2005](https://github.com/Project-HAMi/HAMi/pull/2005) - * feat: add vGPUmonitor --metrics-bind-address flag by @dongjiang1989 in https://github.com/Project-HAMi/HAMi/pull/1613
- * feat: add promtheus serviceMonitor in helm-charts by @dongjiang1989 in https://github.com/Project-HAMi/HAMi/pull/1614
- * feat: add serviceMonitor for device plugin by @dongjiang1989 in https://github.com/Project-HAMi/HAMi/pull/1633
- * feat: check resource quota in webhook by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1605
- * feat: add support for vastai device by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1645
- * feat(helm): add config namespaceSelector & objectSelector for webhook by @haitwang-cloud in https://github.com/Project-HAMi/HAMi/pull/1653
- * feat: add Ascend ResourceCoreName to support hami-vnpu-core virtualization by @ashergaga in https://github.com/Project-HAMi/HAMi/pull/1771
- * feat: add Ascendxxx-core resource by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1804
- * feat: support node filtering based on hami-vnpu-core annotation by @ashergaga in https://github.com/Project-HAMi/HAMi/pull/1812
- * feat: add enableGetPreferredAllocation flag by @DSFans2014 in https://github.com/Project-HAMi/HAMi/pull/1824
- * feat(helm): make device plugin security context configurable via values by @ilia-medvedev in https://github.com/Project-HAMi/HAMi/pull/1887
- * feat(scheduler): support template node simulation filtering by @spencercjh in https://github.com/Project-HAMi/HAMi/pull/2046
- * feat(helm): add switch for admission webhook namespaceSelector by @spencercjh in https://github.com/Project-HAMi/HAMi/pull/2091
- * feat(nvidia): support RTX PRO 6000 Blackwell dynamic MIG by @shkatara in https://github.com/Project-HAMi/HAMi/pull/2135
- * feat(dashboards): add importable Grafana dashboard for HAMi GPU metrics by @imantaba in https://github.com/Project-HAMi/HAMi/pull/2301
- * feat(scheduler): support per-pod device scoring weights by @blackdragoon26 in https://github.com/Project-HAMi/HAMi/pull/2469
- * feat: add hami_host_gpu_memory_controller_utilization_ratio metric by @manoj-1407 in https://github.com/Project-HAMi/HAMi/pull/2616
- * feat(scheduler): export hami_resource_quota_limit gauge metric by @SK8-infi in https://github.com/Project-HAMi/HAMi/pull/2715
- * Support live per-GPU cordon via node annotation by @mohityadav8 in https://github.com/Project-HAMi/HAMi/pull/2298
- * Helm - Render nvidia.overwriteEnv from values with a default of false.
- * Configure ASCEND_VISIBLE_DEVICES env for container and RuntimeClassName for pods
- * device-plugin: opt-in NUMA topology for vGPU replicas
value changes (12)
- - **Change Compile image to ubi8 to support all GLIBC versions** → - [
build(docker): align HAMi with HAMi-core compile imageby @spencercjh #1958](https://github.com/Project-HAMi/HAMi/pull/1958) - [docs: drop stale glibc upper bound from prerequisites#2102](https://github.com/Project-HAMi/HAMi/pull/2102) - * fix: harden HTTP servers and migconfig file permissions by @mesutoezdil in https://github.com/Project-HAMi/HAMi/pull/2100
- * Update HAMi-core and version to v2.10 by @archlitchi in https://github.com/Project-HAMi/HAMi/pull/2759
- * add device type label in metrics
- * vGPUmonitor: skip devices with invalid UTF-8 UUID during container init
- * update nvidia_dp and nvidia_container_runtime module
- * Upgrade Go to 1.26.2
- * bump HAMi-DRA version to v0.2.0
- * build(deps): bump github.com/ccoveille/go-safecast/v2
- * build: bump Go toolchain to 1.26.5
- * chore: bump Go builder to 1.26.6
- * chore: bump golang.org/x/mod to v0.40.0
constraints (9)
- - **Support podgroup** → - [
feat(scheduler): retry NodeLock in Bind for PodGroup membersby @lin121291 #2066](https://github.com/Project-HAMi/HAMi/pull/2066) - [Fix: adopt to gangScheduling feature gates on k8s v1.by @archlitchi #2206](https://github.com/Project-HAMi/HAMi/pull/2206)36+ - - **Flexible Mig** → [
refactor(MIG): support MIG instance dynamic allocation/deallocationby @FouoF #2378](https://github.com/Project-HAMi/HAMi/pull/2378) - - **Scheduler policy combination** → [
feat: support comma-separated gpu-scheduler-policy combinationsby @mesutoezdil #2621](https://github.com/Project-HAMi/HAMi/pull/2621) - - **vNPU Compat mode / Heterogeneous Ascend Mode (vNPU + HAMi-core)** → [
Feat: Allow mode-agnostic Ascend pods to schedule across both vNPU-template and HAMi-core nodesby @ouyangluwei163 #2035](https://github.com/Project-HAMi/HAMi/pull/2035) - * feat: support module-pair allocation for Ascend 910C devices in SuperPod environments by @ashergaga in https://github.com/Project-HAMi/HAMi/pull/1610
- * feat: Supports multi-device requests with hami-vnpu-core enabled. by @ashergaga in https://github.com/Project-HAMi/HAMi/pull/1837
- * feat: Support enflame drs by @FouoF in https://github.com/Project-HAMi/HAMi/pull/1872
- * feat(helm): allow passing device-config content directly via values.yaml by @shkatara in https://github.com/Project-HAMi/HAMi/pull/2277
- * chore: using LeaderElectionConfiguration for kubernetes v1.23 and newer version
removals (1)
- * Remove 'deleted' annotations when registering
renames (1)
- * feat(metrics): align prometheus metric and label names with best practices by @MyoungHaSong in https://github.com/Project-HAMi/HAMi/pull/1644
A weekly email arrives when a release needs action. Like the security patches and breaking changes in this release.