RATATOSKRATATOSK
ログイン

hami

v2.11.0AI & ML
2026年11月4日

ACTION 6CHECK 7PLAN 2OTHER 176

アクセラレーター、スケジューリング、監視、リモート GPU、デプロイ機能を広げる保守リリースで、正確性と性能に関する修正も含まれます。アップグレード前には、変更されたデフォルト値、厳格化された検証と互換性要件、依存関係の更新、非推奨メトリクスの確認が必要です。

要対応 (6)

  • breakingHAMI-core の GPU 全体へのマウント廃止

    GPU 全体の割り当てに HAMI-core をマウントしなくなり、メトリクスは NVML から直接エクスポートされます。

  • breakingGPU 数、メモリ、コア数のデフォルト値を設定

    GPU 数、メモリ、コア数のデフォルト値を設定しました。

  • breakingGPU コア使用率の範囲外要求を拒否

    範囲 [0,100] 外の GPU コア使用率を要求した場合に拒否するようにしました。

  • breakingスケジューラー管理の割り当てアノテーション付き Pod を拒否

    スケジューラーが管理する割り当てアノテーションを付けて作成された Pod を拒否するようにしました。

  • breakingコンテナ要求量を超える割り当てを拒否

    コンテナの要求量を超える割り当てを拒否するようにしました。

  • breaking要求用アノテーションの削除

    要求用アノテーションを削除しました。

影響確認 (7)

  • breaking/refit の呼び出し元認証を必須化

    /refit を使用する場合に適用されます。

    /refit の呼び出し元に対する認証を必須にしました。

  • breakingKubernetes 1.23 を必須化

    Kubernetes 1.23 に依存していない場合に適用されます。

    Kubernetes 1.23 を要件としました。

  • breakingdevice-config のデフォルト値の扱いを変更

    device-config を設定する場合に適用されます。

    device-config のデフォルト値には values.yaml に記載された値だけを使用するようにしました。

  • breakingFit での namespace ResourceQuota 検査

    namespace ResourceQuota を設定する場合に適用されます。

    Fit で namespace ResourceQuota を確認するようにしました。

  • breakinghygon のデフォルトコア値を更新

    hygon を実行する場合に適用されます。

    hygon のデフォルトのコア値を更新しました。

  • breakingbiren と cambricon の不正なデバイス数を拒否

    biren または cambricon を実行する場合に適用されます。

    biren と cambricon で、0 または負のデバイス数を拒否するようにしました。

  • breakingnvidia の小数 GPU メモリ使用率を拒否

    nvidia を実行する場合に適用されます。

    小数の GPU メモリ使用率を拒否するようにしました。

事前準備 (2)

  • deprecatedhami_container_device_memory_bytes の非推奨化

    hami_container_device_memory_bytes を使用する場合に適用されます。

    hami_container_device_memory_bytes を非推奨としました。

  • deprecatedhami_resource_quota_used の limit ラベルを非推奨化

    hami_resource_quota_used を使用する場合に適用されます。

    hami_resource_quota_used メトリクスの limit ラベルを非推奨としました。

その他の記録済み変更 176 件すべてfixes 98 · additions 42 · value changes 23 · constraints 9 · renames 3 · removals 1

fixes (98)

  • Fix/scheduler metrics audit
  • fixing the reject unknown fields in device-config.yml
  • fix(nvidia): honor enableGetPreferredAllocation in GetDevicePluginOptions
  • fix(ascend): guard against empty ResourceMemoryName injection in admission
  • fix(ascend): do not overwrite ASCEND_VISIBLE_DEVICES for Ascend containers
  • fix(chart): use lists for device plugin extraEnvs defaults
  • fix(device-plugin): honor disable-core-limit flag
  • fix(monitor): replace MustNewConstMetric with safe helper
  • [kind/bug] Restore model-aware scheduling for Cambricon MLU devices
  • validate device-scoring-weights at admission
  • validate device resource names and defaults at startup
  • fix: disable MIG when using hami-core
  • guard allocation annotations on update, not just create
  • auto driverroot make mount unchange
  • fail closed on invalid HAMi device requests instead of binding GPU-less pods
  • select Lupine server using all pod container requests
  • surface collection failures
  • run library-copy init first
  • exclude NotReady Lupine servers
  • mount GPU limits at /overrideEnv for login sessions
  • stop mounting the host root for auto driver root
  • requeue pods waiting for a lupine fleet change
  • resolve dynamic MIG identities
  • watch profiling fields
  • Fix vgpu-monitor can't discover mig-instance properly
  • select quota-feasible GPU combinations in topology scheduling
  • serialize expired-lock recovery
  • don't cache node annotation as up-to-date when the patch fails
  • fix(device): reject malformed and negative device allocation annotations
  • fix(nvidia): collapse duplicate MIG candidates in topology scheduling
  • fix(skill): align vGPU metrics summary with actual codebase definitions
  • fix(scheduler): include init container requests in device locking
  • fix(dashboard): align node GPU memory ratio scale
  • fix(vGPUmonitor): gracefully skip GPU utilization metrics when unsupported
  • fix: NUMA refit follow-ups from post-merge review
  • fix(scheduler): keep the configured policy for unrecognized policy an…
  • fix(scheduler): account terminating pods replayed by informer sync
  • fix(scheduler): keep candidate nodes for non HAMi pods on the simulation path
  • fix(metax): let multi container pods spread over exclusive GPUs
  • fix(plugin): pair per device limits by UUID when aligning kubelet IDs
  • fix(awsneuron): cap whole-device core accounting to addressable cores
  • fix(enflame): keep DRS profile core costs within the device budget
  • fix(kunlun): make topology selection independent of device slice order
  • fix(amd): keep allocation annotations decodable for empty registered types
  • fix(device-plugin): remove panic(0) traps from getAPIDevices and dead util code
  • fix(monitor): reuse the node-scoped pod informer for metrics
  • fix(monitor): report metrics server listen failures
  • fix(chart): target the scheduler metrics port by name
  • fix(ascend): gate 910C module pair allocation on SuperPod
  • fix(metrics): drop the constant container index label
  • fix(scheduler): keep bare reasons in FilteringFailed events
  • fix(device-plugin): propagate IMEX channels into the NVIDIA plugin
  • fix(scheduler): use phase-aware accounting during NUMA refit
  • fix(scheduler): validate NUMA refit quota with container positions
  • fix(dashboard): use native node labels for host panels
  • fix(util): return the canonical GPU scheduler policy for padded annotations
  • fix(kunlun): compare elements instead of indices in calcscore sort
  • fix(scheduler): keep multi-slot device usage across annotation round trip
  • fix(monitor): count task priorities in a map, not a slice index
  • fix(chart): render service labels at the right indent level
  • fix(scheduler): clean annotationless pod deletes from cache
  • Fix Ascend webhook reject pod with 'vnpu-core' resource
  • fix(nvidia): guard Allocate against empty DevicesIds requests
  • fix(chart): honor kube-scheduler image pull secrets
  • fix(chart): keep scheduler livenessProbe inside the kubeScheduler container guard
  • fix(scheduler): prune printedLog when a node is deleted
  • fix(chart): keep workload labels on scraped HAMi metrics
  • fix(nvidia): make managed admission env mutation idempotent
  • fix(scheduler): serve the extender verbs on loopback, not the cluster port
  • fix(scheduler): verify pod identity before the extender acts on a request
  • fix(awsneuron): allocate by device index, not slice position
  • fix(iluvatar): read the scaled core limit as a per card value
  • fixing the awsneuron: derive core geometry per node
  • fix(scheduler): score devices against the request on model-typed backends
  • fix(monitor): drop duplicate NVIDIA prefix from device_type label
  • fix(device): enforce minimum limit across multiple ResourceQuota objects
  • fix(remotegpu): keep multi-container pod allocations on one Lupine server
  • fix(scheduler): report undecodable live Pod allocations
  • fix(amd): round core requests up to whole WGPs
  • fix(vGPUmonitor): make the MIG lock wait respect context cancellation
  • fix(scheduler): gpumem score before gpumem-percentage
  • fix(nodelock): keep an expired lock while its holder waits to be allocated
  • fix: defer Ascend memory trimming until node filtering
  • fix(util): serve only the pod the node lock names
  • fix(hygon): prevent dcucores integer overflow
  • fix(amd): judge node health by allocatable GPUs
  • fix(amd): read a registration without a type as the AMD type
  • fix(scheduler): keep HAMI_NODELOCK_EXPIRE unless --node-lock-timeout is set
  • fix(scheduler): restore steady-state usage on pod replay
  • Fix healthcheck always return true for some device vendors
  • guard the PSP and pin images by digest
  • fix(device-plugin): mark all replicas/MIG slices unhealthy on XID
  • fix(device-plugin): check MIG allocations against the registered profiles
  • fix(vGPUmonitor): skip terminated pods in per-container metrics
  • fix(scheduler): ignore negative percentage
  • fix(scheduler): store quota memory limits scaled by memoryFactor
  • fix(scheduler): hold allocLock when removing a pod's allocation
  • fix(chart): add actual NVML names to MIG allowlist

additions (42)

  • PodGroup Support
  • New schedule policy: mutex
  • Add template to vNPU-mode
  • add profiling support with dedicated HTTP bind address
  • expose scheduler kube-qps, kube-burst and kube-timeout as values
  • Flexible-MIG
  • Real-usage monitoring for Flexible-mig workloads
  • Allows running vllm on remote cuda
  • Schedule policy combinations
  • GPU/CPU Numa Align
  • track InitContainers
  • feat: add scheduler NUMA refit for device allocations
  • Add Prometheus metrics for scheduler leader and cache sync state
  • Feat/scheduler poddisruptionbudget
  • feat: add host PID broker server and device plugin integration
  • Expose Hardware Health and Error Metrics in vGPUmonitor
  • add auto detecting nvidiaDriverRoot
  • expose node-lock-retry-timeout and extender httpTimeout
  • allow a pod to prefer a MIG profile over 3g via annotation
  • pod level and container level overwrite env
  • add remote-gpu device backend for lupine-served GPUs
  • configurable per-card memory units
  • Add CDI support for dynamic MIG devices
  • schedule ENPU workloads with policy isolation
  • support amd.com/numa-bind for multi-GPU requests
  • add health probes
  • match nodeconfig by label selector
  • expose Ascend hami-core mode slot capacity
  • set a configurable runtimeClassName on AMD GPU Pods
  • add limitDelivery switch
  • honor device cordon
  • enhancement remote mode
  • introduce deviceSplitCount for both hami-core and enpu mode
  • add a redacted HAMi diagnostic bundle collector
  • select Lupine server pools by label
  • Add 'nvidia-smi' to remote client
  • add MIG monitoring support
  • add securityContext for the lupine server
  • introduce global.imageSource to switch between upstream and CN mirrors
  • support session image pull secrets in configuration and templates
  • add an unknown GPU process count metric
  • add admission.k8s.io/v1 to webhook admissionReviewVersions

value changes (23)

  • libvnpu performance optimization
  • Add Missing Node Label to Memory Controller Metric in vGPUmonitor
  • Optimize the Dynamic MIG instance lifecycle
  • fix(deps): update Go dependencies and bump golang.org/x/net to v0.60.0
  • chore: upgrade to go 1.27
  • chore: build the images with golang 1.27.2
  • score MIG placements and add backtracking fallback
  • stop deep-copying whole Pods into the usage snapshot
  • honor plugin-advertised hami-core percentage budget
  • index device UUIDs once per scrape
  • run GPUs for lupine instead of advertising them locally
  • build the usage snapshot only for candidate nodes
  • report memory and core resources to node capacity
  • honor the topology-aware policy by XGMI links and NUMA
  • whole-GPU hook bypass and direct NVML monitoring
  • enhance fields validator
  • honor preferred nodeAffinity in node score
  • bump Go to 1.27.3
  • pin the dcgm base image
  • bump DCGM to 4.7.0-2
  • go-dcgm to v1.4702.0
  • bump golang.org/x/net from 0.61.0 to 0.62.0
  • Update HAMi-core and image tag

constraints (9)

  • Support Cluster Autoscaler
  • Full AMD Support
  • Ascend 950 Support
  • account native sidecar container GPU resources
  • add OpenShift &SELinux helm chart support
  • whole-card/slice mixing guard
  • support all four NeuronCores on Inferentia1 devices
  • support gpumem-percentage memory requests
  • Configure Ascend950DT 9582 whole-card scheduling

renames (3)

  • rename DCU(HYGON) to HCU across resources, charts, and docs
  • move all vendor configs to devices.VENDOR
  • stable usage.cache name

removals (1)

  • fix(chart): drop unused pods and nodes verbs from the device-plugin ClusterRole
hamiをスタックに追加

対応が必要なリリースが出たときに、週次メールでお知らせします。 今回の破壊的変更も、その一例です。

スタックに追加 →
公式リリースノート ↗hamiのリリース一覧 →