RATATOSKRATATOSK
로그인

hami

v2.11.0AI & ML
2026년 11월 4일

ACTION 6CHECK 7PLAN 2OTHER 176

가속기, 스케줄링, 모니터링, 원격 GPU, 배포 기능을 넓히고 정확성과 성능 문제를 폭넓게 바로잡은 유지보수 릴리스입니다. 업그레이드 전 변경된 기본값과 더 엄격해진 검증 및 호환성 조건, 의존성 변경, 더 이상 권장되지 않는 메트릭을 확인해야 합니다.

조치 필요 (6)

  • breakingHAMI-core 마운트 제거

    전체 GPU 할당에 HAMI-core를 마운트하지 않고, 메트릭을 NVML에서 직접 내보냅니다.

  • breakingGPU 수와 메모리 및 코어 기본값 설정

    GPU 수, 메모리, 코어를 기본값으로 설정합니다.

  • breakingGPU 코어 비율 범위 검증

    GPU 코어 비율이 [0,100] 범위를 벗어난 요청을 거부합니다.

  • breaking스케줄러 소유 할당 주석이 있는 파드 거부

    스케줄러가 소유한 할당 주석이 있는 파드 생성을 거부합니다.

  • breaking컨테이너 요청량 초과 할당 거부

    컨테이너 요청량보다 큰 할당을 거부합니다.

  • breaking요청 주석 제거

    요청 주석을 제거합니다.

영향 확인 (7)

  • breaking/refit 호출자 인증 요구

    /refit을 사용하는 경우에 적용됩니다.

    호출자 인증을 요구합니다.

  • breakingKubernetes 1.23 요구

    Kubernetes 1.23에 의존하지 않는 경우에 적용됩니다.

    Kubernetes 1.23을 요구합니다.

  • breakingdevice-config 기본값 적용 방식 변경

    device-config를 설정하는 경우에 적용됩니다.

    values.yaml의 값만 device-config의 기본값으로 사용합니다.

  • breakingFit의 네임스페이스 ResourceQuota 확인

    네임스페이스 ResourceQuota를 설정한 경우에 적용됩니다.

    Fit에서 네임스페이스 ResourceQuota를 확인합니다.

  • breakinghygon 기본 코어 값 변경

    hygon을 실행하는 경우에 적용됩니다.

    hygon의 기본 코어 값을 업데이트합니다.

  • breaking잘못된 장치 수 요청 거부

    biren 또는 cambricon을 실행하는 경우에 적용됩니다.

    biren과 cambricon에서 장치 수가 0이거나 음수인 요청을 거부합니다.

  • breakingGPU 메모리 소수 비율 요청 거부

    nvidia를 실행하는 경우에 적용됩니다.

    GPU 메모리 비율의 소수값 요청을 거부합니다.

미리 준비 (2)

  • deprecatedhami_container_device_memory_bytes 사용 중단 예고

    hami_container_device_memory_bytes를 사용하는 경우에 적용됩니다.

    hami_container_device_memory_bytes 메트릭을 더 이상 권장하지 않습니다.

  • deprecatedhami_resource_quota_used의 limit 레이블 사용 중단 예고

    hami_resource_quota_used를 사용하는 경우에 적용됩니다.

    hami_resource_quota_used 메트릭의 limit 레이블을 더 이상 권장하지 않습니다.

그 외 기록된 변경 176건 전체fixes 98 · additions 42 · value changes 23 · constraints 9 · renames 3 · removals 1

fixes (98)

  • Fix/scheduler metrics audit
  • fixing the reject unknown fields in device-config.yml
  • fix(nvidia): honor enableGetPreferredAllocation in GetDevicePluginOptions
  • fix(ascend): guard against empty ResourceMemoryName injection in admission
  • fix(ascend): do not overwrite ASCEND_VISIBLE_DEVICES for Ascend containers
  • fix(chart): use lists for device plugin extraEnvs defaults
  • fix(device-plugin): honor disable-core-limit flag
  • fix(monitor): replace MustNewConstMetric with safe helper
  • [kind/bug] Restore model-aware scheduling for Cambricon MLU devices
  • validate device-scoring-weights at admission
  • validate device resource names and defaults at startup
  • fix: disable MIG when using hami-core
  • guard allocation annotations on update, not just create
  • auto driverroot make mount unchange
  • fail closed on invalid HAMi device requests instead of binding GPU-less pods
  • select Lupine server using all pod container requests
  • surface collection failures
  • run library-copy init first
  • exclude NotReady Lupine servers
  • mount GPU limits at /overrideEnv for login sessions
  • stop mounting the host root for auto driver root
  • requeue pods waiting for a lupine fleet change
  • resolve dynamic MIG identities
  • watch profiling fields
  • Fix vgpu-monitor can't discover mig-instance properly
  • select quota-feasible GPU combinations in topology scheduling
  • serialize expired-lock recovery
  • don't cache node annotation as up-to-date when the patch fails
  • fix(device): reject malformed and negative device allocation annotations
  • fix(nvidia): collapse duplicate MIG candidates in topology scheduling
  • fix(skill): align vGPU metrics summary with actual codebase definitions
  • fix(scheduler): include init container requests in device locking
  • fix(dashboard): align node GPU memory ratio scale
  • fix(vGPUmonitor): gracefully skip GPU utilization metrics when unsupported
  • fix: NUMA refit follow-ups from post-merge review
  • fix(scheduler): keep the configured policy for unrecognized policy an…
  • fix(scheduler): account terminating pods replayed by informer sync
  • fix(scheduler): keep candidate nodes for non HAMi pods on the simulation path
  • fix(metax): let multi container pods spread over exclusive GPUs
  • fix(plugin): pair per device limits by UUID when aligning kubelet IDs
  • fix(awsneuron): cap whole-device core accounting to addressable cores
  • fix(enflame): keep DRS profile core costs within the device budget
  • fix(kunlun): make topology selection independent of device slice order
  • fix(amd): keep allocation annotations decodable for empty registered types
  • fix(device-plugin): remove panic(0) traps from getAPIDevices and dead util code
  • fix(monitor): reuse the node-scoped pod informer for metrics
  • fix(monitor): report metrics server listen failures
  • fix(chart): target the scheduler metrics port by name
  • fix(ascend): gate 910C module pair allocation on SuperPod
  • fix(metrics): drop the constant container index label
  • fix(scheduler): keep bare reasons in FilteringFailed events
  • fix(device-plugin): propagate IMEX channels into the NVIDIA plugin
  • fix(scheduler): use phase-aware accounting during NUMA refit
  • fix(scheduler): validate NUMA refit quota with container positions
  • fix(dashboard): use native node labels for host panels
  • fix(util): return the canonical GPU scheduler policy for padded annotations
  • fix(kunlun): compare elements instead of indices in calcscore sort
  • fix(scheduler): keep multi-slot device usage across annotation round trip
  • fix(monitor): count task priorities in a map, not a slice index
  • fix(chart): render service labels at the right indent level
  • fix(scheduler): clean annotationless pod deletes from cache
  • Fix Ascend webhook reject pod with 'vnpu-core' resource
  • fix(nvidia): guard Allocate against empty DevicesIds requests
  • fix(chart): honor kube-scheduler image pull secrets
  • fix(chart): keep scheduler livenessProbe inside the kubeScheduler container guard
  • fix(scheduler): prune printedLog when a node is deleted
  • fix(chart): keep workload labels on scraped HAMi metrics
  • fix(nvidia): make managed admission env mutation idempotent
  • fix(scheduler): serve the extender verbs on loopback, not the cluster port
  • fix(scheduler): verify pod identity before the extender acts on a request
  • fix(awsneuron): allocate by device index, not slice position
  • fix(iluvatar): read the scaled core limit as a per card value
  • fixing the awsneuron: derive core geometry per node
  • fix(scheduler): score devices against the request on model-typed backends
  • fix(monitor): drop duplicate NVIDIA prefix from device_type label
  • fix(device): enforce minimum limit across multiple ResourceQuota objects
  • fix(remotegpu): keep multi-container pod allocations on one Lupine server
  • fix(scheduler): report undecodable live Pod allocations
  • fix(amd): round core requests up to whole WGPs
  • fix(vGPUmonitor): make the MIG lock wait respect context cancellation
  • fix(scheduler): gpumem score before gpumem-percentage
  • fix(nodelock): keep an expired lock while its holder waits to be allocated
  • fix: defer Ascend memory trimming until node filtering
  • fix(util): serve only the pod the node lock names
  • fix(hygon): prevent dcucores integer overflow
  • fix(amd): judge node health by allocatable GPUs
  • fix(amd): read a registration without a type as the AMD type
  • fix(scheduler): keep HAMI_NODELOCK_EXPIRE unless --node-lock-timeout is set
  • fix(scheduler): restore steady-state usage on pod replay
  • Fix healthcheck always return true for some device vendors
  • guard the PSP and pin images by digest
  • fix(device-plugin): mark all replicas/MIG slices unhealthy on XID
  • fix(device-plugin): check MIG allocations against the registered profiles
  • fix(vGPUmonitor): skip terminated pods in per-container metrics
  • fix(scheduler): ignore negative percentage
  • fix(scheduler): store quota memory limits scaled by memoryFactor
  • fix(scheduler): hold allocLock when removing a pod's allocation
  • fix(chart): add actual NVML names to MIG allowlist

additions (42)

  • PodGroup Support
  • New schedule policy: mutex
  • Add template to vNPU-mode
  • add profiling support with dedicated HTTP bind address
  • expose scheduler kube-qps, kube-burst and kube-timeout as values
  • Flexible-MIG
  • Real-usage monitoring for Flexible-mig workloads
  • Allows running vllm on remote cuda
  • Schedule policy combinations
  • GPU/CPU Numa Align
  • track InitContainers
  • feat: add scheduler NUMA refit for device allocations
  • Add Prometheus metrics for scheduler leader and cache sync state
  • Feat/scheduler poddisruptionbudget
  • feat: add host PID broker server and device plugin integration
  • Expose Hardware Health and Error Metrics in vGPUmonitor
  • add auto detecting nvidiaDriverRoot
  • expose node-lock-retry-timeout and extender httpTimeout
  • allow a pod to prefer a MIG profile over 3g via annotation
  • pod level and container level overwrite env
  • add remote-gpu device backend for lupine-served GPUs
  • configurable per-card memory units
  • Add CDI support for dynamic MIG devices
  • schedule ENPU workloads with policy isolation
  • support amd.com/numa-bind for multi-GPU requests
  • add health probes
  • match nodeconfig by label selector
  • expose Ascend hami-core mode slot capacity
  • set a configurable runtimeClassName on AMD GPU Pods
  • add limitDelivery switch
  • honor device cordon
  • enhancement remote mode
  • introduce deviceSplitCount for both hami-core and enpu mode
  • add a redacted HAMi diagnostic bundle collector
  • select Lupine server pools by label
  • Add 'nvidia-smi' to remote client
  • add MIG monitoring support
  • add securityContext for the lupine server
  • introduce global.imageSource to switch between upstream and CN mirrors
  • support session image pull secrets in configuration and templates
  • add an unknown GPU process count metric
  • add admission.k8s.io/v1 to webhook admissionReviewVersions

value changes (23)

  • libvnpu performance optimization
  • Add Missing Node Label to Memory Controller Metric in vGPUmonitor
  • Optimize the Dynamic MIG instance lifecycle
  • fix(deps): update Go dependencies and bump golang.org/x/net to v0.60.0
  • chore: upgrade to go 1.27
  • chore: build the images with golang 1.27.2
  • score MIG placements and add backtracking fallback
  • stop deep-copying whole Pods into the usage snapshot
  • honor plugin-advertised hami-core percentage budget
  • index device UUIDs once per scrape
  • run GPUs for lupine instead of advertising them locally
  • build the usage snapshot only for candidate nodes
  • report memory and core resources to node capacity
  • honor the topology-aware policy by XGMI links and NUMA
  • whole-GPU hook bypass and direct NVML monitoring
  • enhance fields validator
  • honor preferred nodeAffinity in node score
  • bump Go to 1.27.3
  • pin the dcgm base image
  • bump DCGM to 4.7.0-2
  • go-dcgm to v1.4702.0
  • bump golang.org/x/net from 0.61.0 to 0.62.0
  • Update HAMi-core and image tag

constraints (9)

  • Support Cluster Autoscaler
  • Full AMD Support
  • Ascend 950 Support
  • account native sidecar container GPU resources
  • add OpenShift &SELinux helm chart support
  • whole-card/slice mixing guard
  • support all four NeuronCores on Inferentia1 devices
  • support gpumem-percentage memory requests
  • Configure Ascend950DT 9582 whole-card scheduling

renames (3)

  • rename DCU(HYGON) to HCU across resources, charts, and docs
  • move all vendor configs to devices.VENDOR
  • stable usage.cache name

removals (1)

  • fix(chart): drop unused pods and nodes verbs from the device-plugin ClusterRole
hami 스택에 추가

조치가 필요한 릴리스가 나왔을 때 주간 메일로 알려드립니다. 이번 릴리스의 브레이킹 체인지 같은 것들입니다.

스택에 추가 →
공식 릴리스 노트 ↗hami 릴리스 전체 →