Breaking changes27
Four deprecated VTGate metrics removed
AffectsVTGate monitoring dashboards or alerting systems using any of the four removed metrics
The metrics had been deprecated in v22.0.0 and are no longer provided. QueryExecutions replaces QueriesProcessed, QueryRoutes replaces QueriesRouted, and QueryExecutionsByTable replaces both by-table metrics.
Show release-note lines
Four deprecated VTGate metrics have been completely removed in v23.0.0. These metrics were deprecated in v22.0.0:
| Metric Name | Component | Deprecated In |
|---------------------------|-----------|---------------|
| `QueriesProcessed` | vtgate | v22.0.0 |
| `QueriesRouted` | vtgate | v22.0.0 |
| `QueriesProcessedByTable` | vtgate | v22.0.0 |
| `QueriesRoutedByTable` | vtgate | v22.0.0 |
**Impact**: Any monitoring dashboards or alerting systems using these metrics must be updated to use the replacement metrics introduced in v22.0.0:
- Use `QueryExecutions` instead of `QueriesProcessed`
- Use `QueryRoutes` instead of `QueriesRouted`
- Use `QueryExecutionsByTable` instead of `QueriesProcessedByTable` and `QueriesRoutedByTable`
See the [v22.0.0 release notes](https://github.com/vitessio/vitess/blob/main/changelog/22.0/22.0.0/release_notes.md#new-vtgate-metrics) for details on the new metrics.
| Component | Metric Name | Was Deprecated In | Deprecation PR |
|:---------:|:-------------------------:|:-----------------:|:-------------------------------------------------------:|
| `vtgate` | `QueriesProcessed` | `v22.0.0` | [#17727](https://github.com/vitessio/vitess/pull/17727) |
| `vtgate` | `QueriesRouted` | `v22.0.0` | [#17727](https://github.com/vitessio/vitess/pull/17727) |
| `vtgate` | `QueriesProcessedByTable` | `v22.0.0` | [#17727](https://github.com/vitessio/vitess/pull/17727) |
| `vtgate` | `QueriesRoutedByTable` | `v22.0.0` | [#17727](https://github.com/vitessio/vitess/pull/17727) |
github.com/vitessio/vitess/releases/tag/v23.0.0Default MySQL version for vitess/lite:latest changes to 8.4.6
Affectsthe vitess/lite:latest image
Action requiredFor vitess-operator, the one-time 8.0.x to 8.4.x sequence: add innodb_fast_shutdown=0 to the YAML extra cnf and apply it; wait for all pods to be healthy; change the YAML to use the new Docker images; then remove the setting and apply the YAML again.
The tag previously selected MySQL 8.0.40; after upgrading, it selects 8.4.6 without a change to the tag.
Show release-note lines
The default major MySQL version used by our `vitess/lite:latest` image is going from `8.0.40` to `8.4.6`.
This change was merged in [#18569](https://github.com/vitessio/vitess/pull/18569).
* Update MySQL 8.4 to be the default [#18569](https://github.com/vitessio/vitess/pull/18569)
#### Online DDL
* Online DDL: support `alter vitess_migration ... postpone complete` syntax [#18118](https://github.com/vitessio/vitess/pull/18118)
github.com/vitessio/vitess/releases/tag/v23.0.0VTGate now advertises MySQL version 8.4.6 by default
AffectsVTGate deployments
Action requiredIf the desired advertised MySQL version is not 8.4.6, set the mysql_server_version flag to that version.
Previously, VTGate advertised MySQL version 8.0.40; the new default may differ from the MySQL version you are running.
Show release-note lines
VTGate also advertises MySQL version `8.4.6` by default instead of `8.0.40`. If that is not what you are running, you can set the `mysql_server_version` flag to advertise the desired version.
* Update MySQL 8.4 to be the default [#18569](https://github.com/vitessio/vitess/pull/18569)
github.com/vitessio/vitess/releases/tag/v23.0.0VTGate QueryExecutionsByTable counts successful queries only
Affectssetups that collect or use the VTGate QueryExecutionsByTable metric(Inferred)
Previously, failed query attempts were included, so the metric could overstate successful query volume. Failed queries are now tracked separately by error metrics, and values may decrease for workloads with significant failures.
Show release-note lines
#### VTGate QueryExecutionsByTable Behavior Change
The `QueryExecutionsByTable` metric now only counts successful query executions ([PR #18584](https://github.com/vitessio/vitess/pull/18584)). Previously, it counted all query attempts regardless of success/failure.
**Impact**:
- Metric values may decrease if your workload had significant query failures
- More accurate representation of successful query volume
- Failed queries are tracked separately via error metrics
* fix: emit QueryExecutionsByTable for success cases only [#18584](https://github.com/vitessio/vitess/pull/18584)
github.com/vitessio/vitess/releases/tag/v23.0.0VTTablet --skip-user-metrics flag
AffectsVTTablet setups emitting user-based metrics
In environments with many unique users, per-user username labels can cause metric cardinality explosion. When --skip-user-metrics is enabled, the username label is replaced with UserLabelDisabled.
Show release-note lines
- `--skip-user-metrics` flag if enabled, replaces the username label with "UserLabelDisabled" to prevent metric explosion in environments with many unique users ([PR #18085](https://github.com/vitessio/vitess/pull/18085)).
| `--skip-user-metrics` | vttablet | If enabled, replaces the username label in user-based metrics with "UserLabelDisabled" to prevent metric cardinality explosion in environments with many unique users. | [#18085](https://github.com/vitessio/vitess/pull/18085) |
* [vttablet] Emit Constant “userLabelDisabled” When skip-user-metrics is Enabled [#18085](https://github.com/vitessio/vitess/pull/18085)
github.com/vitessio/vitess/releases/tag/v23.0.0Vitess Bullseye Docker images no longer built or published
AffectsVitess Docker images based on debian:bullseye
Builds continue for Debian Bookworm and add Debian Trixie; the default Debian tag remains unchanged.
Show release-note lines
[Bullseye went EOL 1 year ago](https://www.debian.org/releases/), so starting from v23, we will no longer build or publish images based on debian:bullseye ([PR #18609](https://github.com/vitessio/vitess/pull/18609)).
Builds will continue for Debian Bookworm, and add the recently released Debian Trixie. v23 explicitly does not change the default Debian tag to Trixie.
* docker: add trixie and rm bullseye from build matrix [#18609](https://github.com/vitessio/vitess/pull/18609)
github.com/vitessio/vitess/releases/tag/v23.0.0Datadog tracer debug mode now defaults to off
Affectsinstallations using the Datadog tracing plugin(Inferred)
Previously, the plugin always enabled debug mode, producing extremely verbose logging that could reach millions of logs depending on traffic. The --datadog-trace-debug-mode option turns it on.
Evidence“Previously debug mode was hardcoded to true for the tracer in the datadog plugin, which made logging extremely verbose. This can easily add up to millions of logs depending on the amount of traffic your application receives, which can be very costly.” vitessio/vitess/pull/18347 ↗ Show release-note lines
* Configure datadog trace debug mode [#18347](https://github.com/vitessio/vitess/pull/18347)
| `--datadog-trace-debug-mode` | All components | Makes Datadog trace debug mode configurable instead of always-on. Default: false. | [#18347](https://github.com/vitessio/vitess/pull/18347) |
github.com/vitessio/vitess/blob/main/changelog/23.0/23.0.0/changelog.mdImmediate Online DDL cut-over for positive --force-cut-over-after values of <= 1ms
AffectsOnline DDL with a positive --force-cut-over-after value of <= 1ms
Previously, forced cut-over waited until sinceReadyToComplete exceeded the configured value. Now cut-over can happen even when the measured elapsed time is lower.
Evidence“- When `--force-cut-over-after` value is `<= 1ms` we consider it as "immediate" even if we somehow measure the time-since-ready to be less than that.” vitessio/vitess/pull/18423 ↗ Show release-note lines
* Online DDL cutover enhancements [#18423](https://github.com/vitessio/vitess/pull/18423)
github.com/vitessio/vitess/blob/main/changelog/23.0/23.0.0/changelog.mdVTOrc restarts direct replicas for unreachable primaries
AffectsVTOrc-managed replication
For UnreachablePrimary, VTOrc stops and starts replication on one arbitrary direct replica. If only some replicas are failing to replicate, it classifies the condition as UnreachablePrimaryWithBrokenReplicas and restarts all direct replicas; restarts are limited to one per minute per tablet.
Evidence“However, if there is a non-empty subset of replicas that are failing to replicate (and these are **not** all of the replicas) then we analyze as a new `UnreachablePrimaryWithBrokenReplicas`. This scenario is very highly suspicious of really being an early stage for a `DeadPrimary` scenario, and we want to find out sooner rather than later. Here, we restart all replicas.” vitessio/vitess/pull/18626 ↗ Show release-note lines
* `vtorc`: implement `restartDirectReplicas` as response to `UnreachablePrimary` [#18626](https://github.com/vitessio/vitess/pull/18626)
github.com/vitessio/vitess/blob/main/changelog/23.0/23.0.0/changelog.mdDeprecated VTGate metrics and engine primitive methods removed
Affectssetups using deprecated VTGate metrics or engine primitive methods(Inferred)
The removed VTGate metrics include QueriesProcessed.
Evidence“This PR removed the deprecated metrics on VTGate and removed deprecated methods on engine primitive.” vitessio/vitess/pull/18149 ↗ Show release-note lines
* Remove deprecated methods and metrics [#18149](https://github.com/vitessio/vitess/pull/18149)
github.com/vitessio/vitess/blob/main/changelog/23.0/23.0.0/changelog.mdMulti-statement ExecuteFetchAsDba requests rejected
Affectscode or automation submitting multiple semicolon-separated SQL statements to ExecuteFetchAsDba
Action requiredEach statement must be sent in a separate RPC call.
Previously, requests were accepted when every statement was CREATE TABLE or CREATE VIEW; that exception no longer applies.
Show release-note lines
#### <a id="executefetchasdba-multistatement"/>ExecuteFetchAsDba No Longer Accepts Multi-Statement SQL</a>
The `ExecuteFetchAsDba` RPC method in TabletManager now explicitly rejects SQL queries containing multiple statements (as of [PR #18183](https://github.com/vitessio/vitess/pull/18183)).
**Impact**: Code or automation that previously passed multiple semicolon-separated SQL statements to `ExecuteFetchAsDba` will now receive an error. Each SQL statement must be sent in a separate RPC call.
**Migration**: Split multi-statement SQL into individual RPC calls:
```go
// Before (no longer works):
ExecuteFetchAsDba("CREATE TABLE t1 (id INT); CREATE TABLE t2 (id INT);")
// After (required in v23+):
ExecuteFetchAsDba("CREATE TABLE t1 (id INT);")
ExecuteFetchAsDba("CREATE TABLE t2 (id INT);")
```
* tablet manager's `ExecuteFetchAsDba` does not allow multiple queries [#18183](https://github.com/vitessio/vitess/pull/18183) github.com/vitessio/vitess/releases/tag/v23.0.0Correct error codes from the vttablet tabletmanager client
Affectssetups using the vttablet gRPC tabletmanager client
Action requiredExternal automation relying on google-gRPC error codes must be updated to inspect codes with vterrors.Code(err) and use vtrpcpb.Code values, such as vtrpcpb.Code_NOT_FOUND.
When a tabletmanager RPC failed to execute—for example, because mysqld was down—the client could report Unknown; now that case reports Unavailable, with the error message unchanged.
Evidence“Now tablets with downed-`mysqld` will return the gRPC `Unavailable` code with the error message unchanged, for example” vitessio/vitess/pull/18565 ↗ Show release-note lines
#### <a id="grpc-error-codes"/>gRPC TabletManager Error Code Changes</a>
The `vttablet` gRPC `tabletmanager` client now returns errors wrapped by the internal `go/vt/vterrors` package ([PR #18565](https://github.com/vitessio/vitess/pull/18565)).
**Impact**: External automation relying on google-gRPC error codes must be updated to use `vterrors.Code(err)` to inspect error codes, which returns `vtrpcpb.Code`s defined in [proto/vtrpc.proto](https://github.com/vitessio/vitess/blob/main/proto/vtrpc.proto#L60).
**Migration**:
```go
// Before:
if status.Code(err) == codes.NotFound { ... }
// After:
if vterrors.Code(err) == vtrpcpb.Code_NOT_FOUND { ... }
```
#### <a id="grpctmclient-err-changes"/>gRPC `tabletmanager` client error changes</a>
The `vttablet` gRPC `tabletmanager` client now returns errors wrapped by the internal `go/vt/vterrors` package ([PR #18565](https://github.com/vitessio/vitess/pull/18565)). External automation relying on google-gRPC error codes should now use `vterrors.Code(err)` to inspect the code of an error, which returns `vtrpcpb.Code`s defined in [the `proto/vtrpc.proto` protobuf](https://github.com/vitessio/vitess/blob/main/proto/vtrpc.proto#L60).
See [Breaking Changes](#grpc-error-codes) for migration guidance.
* `tabletmanager`/`grpctmclient`: use `vtrpcpb`/gRPC error codes [#18565](https://github.com/vitessio/vitess/pull/18565)github.com/vitessio/vitess/releases/tag/v23.0.0GTID parsing method adds commit and sequence values
Affectscode calling the GTID parsing APIs directly
Action requiredUpdates to direct callers of the GTID parsing APIs to accommodate the changed method signature
Previously, BinlogEvent.GTID() returned only the GTID, hasBegin, and an error; its signature now also includes lastCommitted and sequenceNumber.
Show release-note lines
Several GTID-related API signatures changed in [PR #18196](https://github.com/vitessio/vitess/pull/18196) as part of GTID performance optimizations:
**Changed**: `BinlogEvent.GTID()` method signature
**Impact**: Code directly using the GTID parsing APIs may need updates. Most users are unaffected as these are internal APIs.
github.com/vitessio/vitess/releases/tag/v23.0.0key.GenerateShardRanges() now accepts a hex-width parameter
Affectscode that calls key.GenerateShardRanges() directly
Action requiredDirect callers must update their calls to pass the new parameter.
Previously, callers supplied only the shard count. The new hexChars value controls the hex width of generated shard names.
Show release-note lines
#### <a id="generateshardranges-api"/>GenerateShardRanges API Signature Change</a>
The `key.GenerateShardRanges()` function signature changed in [PR #18633](https://github.com/vitessio/vitess/pull/18633) to add a new `hexChars int` parameter controlling the hex width of generated shard names.
**Impact**: Code calling `GenerateShardRanges()` directly must be updated to pass the new parameter.
---
github.com/vitessio/vitess/releases/tag/v23.0.0Vitess query buffering during eligible DDL changes
Affectseligible INSTANT DDL migrations and range partition rotation changes
Action requiredAn instant-able migration with --prefer-instant-ddl, or a range partition rotation change.
This reduces query failures while the schema change runs and is transparent to applications.
Show release-note lines
Query buffering has been implemented for `INSTANT` DDL operations ([PR #17945](https://github.com/vitessio/vitess/pull/17945)), reducing query failures during schema changes.
- Automatic buffering of queries during INSTANT DDL execution
- Transparent to applications
**Impact**: Applications experience fewer query errors during schema changes, improving availability during DDL operations.
* Query buffering, terminating blocking transactions for `INSTANT` DDL and other "special plans" [#17945](https://github.com/vitessio/vitess/pull/17945)
github.com/vitessio/vitess/releases/tag/v23.0.0Lock-holder termination before eligible special-operation DDL
Affectseligible instant DDL and range partition rotation operations with nonzero --force-cut-over-after
Previously, these operations ran the DDL directly. Now, queries and transactions holding locks or metadata locks on the migrated table are terminated before ALTER TABLE; the flag's value does not change whether this happens, and termination is skipped while an atomic transaction is pending for the table.
Evidence“In addition, if `--force-cut-over-duration` flag is nonzero, and irrespective of its value, we also issue a one time termination for queries and transactions holding locks or metadata locks on the migrated table, just before issuing the `ALTER TABLE` statement.” vitessio/vitess/pull/17945 ↗ Show release-note lines
- Forced termination of blocking transactions
* Query buffering, terminating blocking transactions for `INSTANT` DDL and other "special plans" [#17945](https://github.com/vitessio/vitess/pull/17945)
github.com/vitessio/vitess/releases/tag/v23.0.0Cgroup CPU and memory metrics now require cgroup v2
AffectsLinux installations using cgroup v1
Previously, Vitess collected CPU and memory metrics from both cgroup v1 and v2. With cgroup v1 or failed cgroup metric setup, it now warns and uses host metrics; cgroup v2 continues to use cgroup metrics.
Evidence“func getCgroup1CpuUsage() (uint64, error) {” vitessio/vitess/pull/18801 ↗ Show release-note lines
#### Observability
* [release-23.0] Improve cgroup metric management (#18791) [#18801](https://github.com/vitessio/vitess/pull/18801)
github.com/vitessio/vitess/blob/main/changelog/23.0/23.0.0/changelog.mdVStream diagnostic logging changes
Affectsclusters using VStream(Inferred)
The added lines are intended to help localize frequent EOF errors in a cluster. One VStream error log changes from Errorf to Infof.
Show release-note lines
* VStream API: add additional log lines for debugging frequent eof errors during catchup [#18124](https://github.com/vitessio/vitess/pull/18124)
github.com/vitessio/vitess/blob/main/changelog/23.0/23.0.0/changelog.mdVReplication stops indefinitely retrying MySQL error 1419
AffectsVReplication workflows
The error can occur when a workflow creates a DEFINER trigger; before this change, VReplication did not handle it and could retry the workflow indefinitely.
Evidence“name: "SQL error with ERBinlogCreateRoutineNeedSuper"” vitessio/vitess/pull/18820 ↗ Show release-note lines
* [release-23.0] VReplication: Treat ER_BINLOG_CREATE_ROUTINE_NEED_SUPER as unrecoverable (#18784) [#18820](https://github.com/vitessio/vitess/pull/18820)
github.com/vitessio/vitess/blob/main/changelog/23.0/23.0.0/changelog.mdVTTablet reports stale replica heartbeats as errors
Affectsreplica tablets using the heartbeat reader(Inferred)
When MySQL stalls and no heartbeat has been successfully read for more than two configured heartbeat intervals, vttablet could keep reporting MySQL as healthy. Now the heartbeat status returns an error and signals VTGate that the tablet is unhealthy.
Evidence“The workaround I'm proposing is to change `*heartbeatReader.Status` func to just disregard any previously read heartbeat data if the last time the heartbeat was successfully read is older than two times the configured interval. It feels a bit hacky, but works reliably in identifying that MySQL is unhealthy and very quickly signals to `vtgate` that the tablet is unhealthy.” vitessio/vitess/pull/17883 ↗ Show release-note lines
* Fix `vttablet` not being marked as not serving when MySQL stalls [#17883](https://github.com/vitessio/vitess/pull/17883)
github.com/vitessio/vitess/blob/main/changelog/23.0/23.0.0/changelog.mdInitAuthServerClientCert now takes an explicit CA value
AffectsGo code calling go/mysql's public InitAuthServerClientCert function, including the VTGate plugin initializer(Inferred)
Previously, the function looked up mysql-server-ssl-ca, falling back to mysql_server_ssl_ca when the first flag was empty. It now takes the CA value as a second argument, which the VTGate plugin initializer supplies through vtgate.GetMysqlServerSSLCA().
Evidence“func InitAuthServerClientCert(clientcertAuthMethod string, caValue string) {” vitessio/vitess/pull/18508 ↗ Show release-note lines
* Set up better dependency checks [#18508](https://github.com/vitessio/vitess/pull/18508)
github.com/vitessio/vitess/blob/main/changelog/23.0/23.0.0/changelog.mdAdded lite:mysql80 Docker image build
Affectsbuilds selecting the lite:mysql80 image(Inferred)
Previously, the lite image build matrix included latest, mysql84, and percona80, but not mysql80.
Show release-note lines
* build `lite:mysql80` docker image [#18692](https://github.com/vitessio/vitess/pull/18692)
github.com/vitessio/vitess/blob/main/changelog/23.0/23.0.0/changelog.mdForced cut-over now terminates table lock holders during RENAME
AffectsOnline DDL migrations using forced cut-over
Previously, a long-running query could start between the pre-cut-over termination and the RENAME, leaving it holding a lock on the migrated table. The additional termination pass excludes the lock and rename connections.
Evidence“With forced cut-over, we now kill queries and transactions holding onto locks on migrated table, not only _before_ the cut-over, but also while the `RENAME` is being applied. Without this, there's a race condition where a long running query could start running just after queries are killed and right before the `RENAME` starts running.” vitessio/vitess/pull/18423 ↗ Show release-note lines
* Online DDL cutover enhancements [#18423](https://github.com/vitessio/vitess/pull/18423)
github.com/vitessio/vitess/blob/main/changelog/23.0/23.0.0/changelog.mdOnline DDL cut-over lock-wait timeouts shortened
AffectsOnline DDL cut-over migrations(Inferred)
The locking connection's timeout changes from five times onlineDDL.CutOverThreshold to three times the threshold. The rename connection's timeout changes from five times the threshold times four to twice the threshold.
Evidence“- Sanitized and reduced query and lock timeouts during the cut-over. There were some excessive timeouts, notably an overlooked `5*onlineDDL.CutOverThreshold*4` value which evaluates to `5min` on a `15s` timeout.” vitessio/vitess/pull/18423 ↗ Show release-note lines
* Online DDL cutover enhancements [#18423](https://github.com/vitessio/vitess/pull/18423)
github.com/vitessio/vitess/blob/main/changelog/23.0/23.0.0/changelog.mdVTOrc database_instance location fields replaced by informational cell
AffectsVTOrc installations using the database_instance table(Inferred)
The removed fields were not used in logic; physical_environment and region were always undefined or empty, while data_center matched the tablet cell. The new cell field is not used in logic yet.
Evidence“The 3 x fields are replaced by the new `cell` field. It isn't used in any logic yet - for now it is informational” vitessio/vitess/pull/18339 ↗ Show release-note lines
* `vtorc`: cleanup `database_instance` location fields [#18339](https://github.com/vitessio/vitess/pull/18339)
github.com/vitessio/vitess/blob/main/changelog/23.0/23.0.0/changelog.mdVTOrc database_instance schema replaces location fields with cell
AffectsVTOrc installations using the database_instance table(Inferred)
Previously, the table had data_center, physical_environment, and region fields, which the pull request says were not used in logic. The replacement cell field is informational and is not used in logic yet.
Evidence“The 3 x fields are replaced by the new `cell` field. It isn't used in any logic yet - for now it is informational” vitessio/vitess/pull/18339 ↗ Show release-note lines
* `vtorc`: cleanup `database_instance` location fields [#18339](https://github.com/vitessio/vitess/pull/18339)
github.com/vitessio/vitess/blob/main/changelog/23.0/23.0.0/changelog.mdVTOrc's aggregated discovery metrics API removed
AffectsIntegrations using VTOrc's /api/aggregated-discovery-metrics endpoint(Inferred)
The endpoint previously returned JSON of aggregated discovery samples over a 120-second period.
Evidence“This PR drops the undocumented VTOrc API endpoint `/api/aggregated-discovery-metrics`” vitessio/vitess/pull/18672 ↗ Show release-note lines
* `vtorc`: remove aggregated discovery metrics HTTP API, cleanup code [#18672](https://github.com/vitessio/vitess/pull/18672)
#### <a id="aggregated-discovery-metrics-api-removed"/>Aggregated Discovery Metrics HTTP API removed</a>
VTOrc's undocumented `/api/aggregated-discovery-metrics` HTTP API endpoint was removed ([PR #18672](https://github.com/vitessio/vitess/pull/18672)). The list of documented VTOrc APIs can be found [here](https://vitess.io/docs/current/reference/vtorc/ui_api_metrics/#apis).
We recommend using the standard VTOrc metrics to gather the same metrics. If you find that a metric is missing in standard metrics, please open an issue or PR to address this.
github.com/vitessio/vitess/blob/main/changelog/23.0/23.0.0/changelog.md