RATATOSKRATATOSK
Sign in

Karmada

v1.16.9Orchestration & Management
Aug 31, 2026

ACTION 3CHECK 1PLAN 2OTHER 105

A maintenance release that expands multi-component scheduling, eviction and replica management, interpreter support, APIs, and observability while improving controller performance and correctness. It also updates Go, Alpine, and other dependencies, including fixes for disclosed security issues.

Action needed (3)

  • securitymediumgithub.com/vektra/mockery update for GO-2025-3900

    The dependency github.com/vektra/mockery was updated to v3.5.5 to address security concerns associated with GO-2025-3900.

  • securityThe alpine base image update

    The base image alpine was updated from alpine:3.23.4 to alpine:3.24.1 to address security concerns.

    karmada#7628karmada#7652

  • securityThe alpine base image update

    The base image alpine was updated from alpine:3.23.3 to alpine:3.23.4 to address security concerns.

    karmada#7410

Check if affected (1)

  • breakingRemoval of external etcd fields from karmada-operator

    Applies if you configure any of the external etcd fields CAData, CertData, or KeyData.

    karmada-operator no longer includes the deprecated external etcd fields CAData, CertData, and KeyData.

Plan ahead (2)

  • deprecatedThe --etcd-init-image flag, deprecatedremoval date not announced

    Applies if you use the --etcd-init-image flag of the init command.

    In karmadactl, the --etcd-init-image flag for the init command is deprecated because it is no longer used and is scheduled for removal.

  • breakingThe member_cluster Prometheus metric labelremoval planned in 1.18

    Applies if you use either the cluster or cluster_name Prometheus metric label.

    The Prometheus metric labels cluster and cluster_name are deprecated for identifying a Karmada member cluster. The member_cluster label is used instead.

All 105 other recorded changesfixes 43 · additions 32 · value changes 18 · constraints 12

fixes (43)

  • - karmada-controller-manager: Fixed an issue where the taint-manager eviction queue would enqueue bindings with indefinite taint tolerations. ([#7775](https://github.com/karmada-io/karmada/pull/7775), @karmada-bot)
  • - karmada-controller-manager: Fixed an issue where Cluster.status.remedyActions could remain stale after an associated Remedy resource was removed. ([#7790](https://github.com/karmada-io/karmada/pull/7790), @karmada-bot)
  • - helm chart: Fixed TLS certificate SAN mismatch when deploying to a custom namespace by adding systemNamespace SANs to certs.auto.hosts. ([#7686](https://github.com/karmada-io/karmada/pull/7686), @karmada-bot)
  • - karmada-search: Fixed the issue that watch connect cannot reflect resources from recovered clusters immediately. ([#7522](https://github.com/karmada-io/karmada/pull/7522), @Ady0333)
  • - karmada-scheduler: Fixed the issue when cluster resources are insufficient, multiple template resources can still be scheduled. ([#7580](https://github.com/karmada-io/karmada/pull/7580), @jabellard)
  • - karmada-controller-manager: Fixed the issue that a transient ClusterClientSetFunc failure (e.g. missing SecretRef during credential rotation) would immediately set the cluster Ready=False without respecting ClusterFailureThreshold, potentially triggering unnecessary workload failover. ([#7578](https://github.com/karmada-io/karmada/pull/7578), @driegel1)
  • - karmada-operator: Fixed init reconciliation failure by replacing non-idempotent secret creation with an idempotent approach. ([#7405](https://github.com/karmada-io/karmada/pull/7405), @anr)
  • - karmada-scheduler: Fixed an issue where the schedule success event was missing cluster information when scheduling with ClusterAffinities. ([#7419](https://github.com/karmada-io/karmada/pull/7419), @cotishq)
  • - karmada-scheduler: Fixed incorrect error type propagation that caused bindings with insufficient cluster replicas to be misrouted to backoffQ instead of unschedulableBindings. ([#7354](https://github.com/karmada-io/karmada/pull/7354), @SujoyDutta)
  • - Fixed the issue that Job completions were assigned to the wrong replicas for each cluster. ([#7401](https://github.com/karmada-io/karmada/pull/7401), @Ady0333)
  • - karmada-agent: Fixed the issue where certificate rotation CSRs were never auto-approved due to a SignerName mismatch between cert_rotation_controller and agent_csr_approving. ([#7310](https://github.com/karmada-io/karmada/pull/7310), @Denyme24)
  • - karmada-chart: Fixed unrendered {{ ca_crt }} during upgrades. ([#7330](https://github.com/karmada-io/karmada/pull/7330), @AbhinavPInamdar)
  • - karmada-controller-manager: Fixed a race condition where graceful eviction tasks could be silently dropped when multiple controllers concurrently modify the same ResourceBinding or ClusterResourceBinding, preventing workloads from being evacuated from tainted or failing clusters. ([#7307](https://github.com/karmada-io/karmada/pull/7307), @Ady0333)
  • - karmada-controller-manager: Fixed the issue where the job status aggregator could enter an error loop due to a race condition when setting the initial startTime. ([#7158](https://github.com/karmada-io/karmada/pull/7158), @rohan-019)
  • - karmada-controller-manager: Fixed CronFederatedHPA scale-up from zero failure when the replicas field is missing. ([#7212](https://github.com/karmada-io/karmada/pull/7212), @zhengjr9)
  • - karmada-controller-manager: Fixed an issue where a per-task GracePeriodSeconds value could leak to subsequent graceful eviction tasks, causing premature or delayed evictions. ([#7187](https://github.com/karmada-io/karmada/pull/7187), @Ady0333)
  • - karmada-controller-manager: Fixed an issue where dependency updates could overwrite other controller annotations during retry conflicts. ([#7216](https://github.com/karmada-io/karmada/pull/7216), @Ady0333)
  • - karmada-scheduler: Fixed a scheduler panic caused by a divide-by-zero error when calculating spread constraints with no valid clusters. ([#7234](https://github.com/karmada-io/karmada/pull/7234), @XiShanYongYe-Chang)
  • - karmada-scheduler: Fixed the bug in the backoff queue where the sorting function was incorrect, potentially causing high-priority items with long backoffs to block lower-priority items. ([#7231](https://github.com/karmada-io/karmada/pull/7231), @zhzhuang-zju)
  • - karmada-scheduler-estimator: Fixed the issue where the resource quota plugin failed to list resource quotas due to a missing namespace in the gRPC request. ([#7238](https://github.com/karmada-io/karmada/pull/7238), @zhzhuang-zju)
  • - karmada-controller-manager: Fixed an issue where policy deletion could be blocked if a resource selector targeted a non-existent resource. ([#7083](https://github.com/karmada-io/karmada/pull/7083), @FAUST-BENCHOU)
  • - karmada-scheduler: Fixed bug preventing multi-component workloads from being rescheduled during cluster failover events. ([#7130](https://github.com/karmada-io/karmada/pull/7130), @mszacillo)
  • - karmada-webhook: Fixed an issue where the condition.reason was not set to QuotaExceeded when FederatedResourceQuota is exceeded. ([#7098](https://github.com/karmada-io/karmada/pull/7098), @kajal-jotwani)
  • - karmadactl: Fixed the messy auto-completion suggestions for commands like 'get' and 'apply'. ([#7025](https://github.com/karmada-io/karmada/pull/7025), @zhzhuang-zju)
  • - karmada-controller-manager: Fixed the issue where PP/CPP cannot be deleted because the resources API selected by the PP/CPP do not exist on the control plane. ([#7028](https://github.com/karmada-io/karmada/pull/7028), @XiShanYongYe-Chang)
  • - karmada-controller-manager: Fixed the issue that HelmRelease did not define observedGeneration variable in the statusAggregation operation. ([#7059](https://github.com/karmada-io/karmada/pull/7059), @FAUST-BENCHOU)
  • karmada-controller-manager: Fixed the issue that rbSpec.Components is not updated when the template is updated.
  • karmada-controller-manager: Added a default timeout (32s) for the member cluster client to avoid the controller hanging when the member cluster does not respond.
  • karmada-controller-manager: Fixed the Job status cannot be aggregated issue due to the missing JobSuccessCriteriaMet condition when using kube-apiserver v1.32+ as Karmada API server.
  • karmada-controller-manager: Fixed the issue that attached resource changes were not synchronized to the cluster in the dependencies distributor.
  • karmada-metrics-adapter: Fixed a panic when querying node metrics by name caused by using the wrong GroupVersionResource (PodsGVR instead of NodesGVR) when creating a lister.
  • karmada-operator: Fixed the issue that CRDs cannot be updated during upgrades of the Karmada instance.
  • karmada-scheduler: Fixed the issue where increasing the total number of replicas can cause some clusters to receive fewer replicas under the StaticWeight strategy by introducing the Webster algorithm.
  • karmada-webhook: Fixed the issue that resourcebinding validating webhook may panic when ReplicaRequirements of a Component in rbSpec.Components is nil.
  • karmadactl: Fixed the issue that the register command still uses the cluster-info endpoint when registering a pull-mode cluster, even if the user provides the API server endpoint.
  • ResourceInterpreter: Fixed the issue that when an object API field name contains dots or colons, it would cause the resource interpreter to fail.
  • - karmada-metrics-adapter: Fixed a panic when querying node metrics by name caused by using the wrong GroupVersionResource (PodsGVR instead of NodesGVR) when creating a lister.
  • - karmadactl: Fixed the issue that the register command still uses the cluster-info endpoint when registering a pull-mode cluster, even if the user provides the API server endpoint.
  • - karmada-scheduler: Fixed the issue where increasing the total number of replicas can cause some clusters to receive fewer replicas under the StaticWeight strategy by introducing the Webster algorithm.
  • - karmada-controller-manager: Fixed the issue that rbSpec.Components is not updated when the template is updated.
  • - ResourceInterpreter: Fixed the issue that when an object API field name contains dots or colons, it would cause the resource interpreter to fail.
  • - karmada-webhook: Fixed the issue that resourcebinding validating webhook may panic when ReplicaRequirements of a Component in rbSpec.Components is nil.
  • - karmada-operator: Fixed the issue that CRDs can not be updated during upgrades of the Karmada instance.

additions (32)

  • In Karmada v1.16.0, we introduce **multi-component scheduling**, a new capability that enables the **complete and unified placement of multi-component workloads**—those composed of multiple interrelated components (e.g., jobManager and taskManagers of FlinkDeployment)—**into a single member cluster with sufficient resources**.
  • - Estimate how many full sets of components can fit into a member cluster based on **ResourceQuota** limits.
  • - Predict schedulability using **actual node resource availability** across clusters. Karmada v1.16.0 ships with built-in **resource interpreters** for the following multi-component workload types:
  • This release introduces an eviction queue with rate limiting capabilities for the Karmada taint manager.
  • - **Configurable Fixed Rate Limiting**: Configure the eviction rate per second through the --eviction-rate command-line flag.
  • - **Comprehensive Metrics Support**: Provides metrics for queue depth, resource kind, processing latency, and success/failure rates for monitoring and troubleshooting.
  • Introduced Components field to ResourceInterpreterContext in the ResourceInterpreterResponse to support interpreting components for webhook interpreter.
  • Introduced a PodDisruptionBudget field to the CommonSettings of Karmada API for supporting PodDisruptionBudgets (PDBs) for Karmada control plane components.
  • karmadactl: The init command now supports customizing Karmada component command line flags.
  • karmadactl: The init command now supports customizing Karmada component command line flags via the configuration file.
  • karmada-controller-manager: Introduced built-in interpreter for Volcano Job.
  • karmada-controller-manager: Introduced a built-in interpreter for Kubeflow Notebooks.
  • karmada-controller-manager: Introduced built-in interpreter for SparkApplication.
  • karmada-controller-manager: Introduced built-in interpreter for PyTorchJob.
  • karmada-controller-manager: Introduced built-in resource interpreter for Kubernetes ReplicaSet workloads.
  • karmada-controller-manager: Introduced built-in interpreter for MPIJob.
  • karmada-controller-manager: Introduced --resource-eviction-rate flag to specify the eviction rate during cluster failover.
  • karmada-scheduler: Implemented MaxAvailableComponentSets interface for general estimator based on resource summary.
  • karmada-scheduler: Enabled the capability for multiple component estimation in the scheduler. The feature is gated behind MultiplePodTemplatesScheduling.
  • karmada-scheduler-estimator: Introduced MaxAvailableComponentSetsRequest & MaxAvailableComponentSetsResponse for component scheduling.
  • karmada-scheduler-estimator: Added plugins in estimator for component scheduling.
  • karmada-scheduler-estimator: Added ResourceQuota plugin for multi-component scheduling.
  • karmada-scheduler-estimator: Implemented the noderesource plugin for multi-component scheduling estimation.
  • karmada-webhook: Enabled federated resource quota calculation for multi-component scheduling.
  • ResourceInterpreter: Enabled GetComponents interpreter operation through Webhook Interpreter.
  • ResourceInterpreter: Added maxAvailableComponentSets to estimator interface.
  • karmada-controller-manager: Added a new Warning event DependencyPolicyConflict to surface when dependency policies have conflicts.
  • karmada-controller-manager: Added new metrics for the failover eviction queue to enhance observability.
  • - karmada-scheduler-estimator: Introduce MaxAvailableComponentSetsRequest & MaxAvailableComponentSetsResponse for component scheduling.
  • - karmada-scheduler: Implemented MaxAvailableComponentSets interface for general estimator based on resource summary.
  • - ResourceInterpreter: Adding maxAvailableComponentSets to estimator interface.
  • - Helm chart: Added helm index for 1.15 release.

value changes (18)

  • - The base image alpine has now been promoted from alpine:3.23.2 to alpine:3.23.3. ([#7162](https://github.com/karmada-io/karmada/pull/7162), @dependabot)
  • - The base image alpine has been promoted from alpine:3.22.2 to alpine:3.23.0. ([#7004](https://github.com/karmada-io/karmada/pull/7004), @dependabot)
  • - The base image alpine has been promoted from alpine:3.23.0 to alpine:3.23.2. ([#7037](https://github.com/karmada-io/karmada/pull/7037), @dependabot)
  • - **Monotonic replica assignment**: Increasing the total replica count will never cause any cluster to lose replicas, ensuring consistent and intuitive behavior.
  • - **Fair handling of remainder replicas**: When distributing replicas among clusters with equal weights, priority is given to the cluster with fewer current replicas. This "smaller-first" approach promotes balanced deployment and better satisfies high availability (HA) requirements.
  • karmadactl: The init command's default kube-apiserver and kube-controller-manager images have been updated from v1.31.3 to v1.34.1. And the default etcd image has been updated from 3.5.16-0 to 3.6.0-0.
  • karmada-controller-manager: Computed effective field values for attached ResourceBindings when referenced by multiple ResourceBindings.
  • karmada-scheduler: Migrated dynamic weight assignment to use the Webster algorithm.
  • karmada-scheduler-estimator: Implemented maxAvailableComponentSets for the accurate estimator.
  • Karmada is now built with Golang v1.24.10.
  • Kubernetes dependencies have been updated to v1.34.1.
  • Updated sigs.k8s.io/controller-runtime from v0.21.0 to v0.22.4.
  • The base image alpine has been promoted from 3.22.1 to 3.22.2.
  • karmada-controller-manager: After enabling the ControllerPriorityQueue feature gate, the asyncWorker uses a priority queue based implementation, which affects the processing order of items in the resource detector and causes the monitoring metric workqueue_depth to be split into multiple series.
  • - Karmada is now built with Golang v1.24.9.
  • - The base image alpine now has been promoted from 3.22.1 to 3.22.2.
  • - Introduced Components field to ResourceInterpreterContext in the ResourceInterpreterResponse to support interpreting components for webhook interpreter.
  • we introduce the Webster method, also known as the Sainte-Laguë method, to improve replica assignment during cross-cluster scheduling.

constraints (12)

  • - FlinkDeployment (flink.apache.org/v1beta1/FlinkDeployment)
  • - SparkApplication (sparkoperator.k8s.io/v1beta2/SparkApplication)
  • - Volcano Job (batch.volcano.sh/v1alpha1/Job)
  • - MPIJob (kubeflow.org/v2beta1/MPIJob)
  • - RayCluster (ray.io/v1/RayCluster)
  • - RayJob (ray.io/v1/RayJob)
  • - TFJob (kubeflow.org/v1/TFJob)
  • **Note**: Multi-component scheduling is **disabled by default**. To use it, enable the MultiplePodTemplatesScheduling > feature gate in your Karmada control plane components.
  • For controllers not built on controller-runtime, such as the detector controller, we extend this capability in release-1.16 by enabling the priority-queue functionality for all controllers that use async workers.
  • karmada-controller-manager: The ControllerPriorityQueue feature gate now applies to all controllers, including async workers. Enable it with --feature-gates=ControllerPriorityQueue=true.
  • - karmadactl: The init command now supports customizing Karmada component command line flags.
  • - ResourceInterpreter: Enable GetComponents interpreter operation through Webhook Interpreter.
Add Karmada to your stack

A weekly email arrives when a release needs action. Like the security patches and breaking changes in this release.

Add to stack →
Official release note ↗All Karmada releases →