Kubernetes v1.36 brings a major milestone for cluster upgrade safety: the Mixed Version Proxy (MVP) is now Beta and enabled by default. Originally introduced as an Alpha feature in v1.28, MVP solves a persistent headache during multi‑version control plane upgrades—preventing incorrect 404 errors when a request lands on an API server that doesn't yet serve a newer resource. Over the past few releases, the feature has undergone a significant architectural overhaul, replacing older discovery mechanisms with modern, dynamic approaches. In this article, we break down the five essential things you need to know about MVP in v1.36, from the core problem it solves to how its evolution makes your clusters more resilient.
1. The Core Problem: Eliminating False 404 Errors During Upgrades
When upgrading a highly available Kubernetes control plane, API servers often run different versions concurrently. Each server may serve a different set of API groups, versions, or resources. Without MVP, a client request for a resource (say, a new API version introduced in the upgrade) that hits an older server would receive a 404 Not Found—even though the resource exists elsewhere in the cluster. This false negative can trigger serious side effects, such as mistaken garbage collection of objects or blocked namespace deletions. MVP solves this by transparently proxying such requests to a peer API server that can serve them, ensuring clients always get the correct response. The feature acts as a safety net, making rolling upgrades far less disruptive for operators and controllers alike.
2. From StorageVersion to Aggregated Discovery: A Smarter Peer Lookup
In its Alpha incarnation, MVP relied on the StorageVersion API to determine which peer API servers could serve which resources. While functional, this approach had a critical limitation: the StorageVersion API is not supported for Custom Resource Definitions (CRDs) or aggregated APIs. This meant MVP could not protect against false 404s for those resource types, leaving a significant gap. For the Beta release in v1.36, the team replaced the StorageVersion dependency with Aggregated Discovery. Now, each API server uses aggregated discovery data to dynamically learn the full capability set of its peers—including CRDs and aggregated APIs. This change makes MVP far more comprehensive and future‑proof, as it no longer hinges on a narrow discovery mechanism.
3. Peer‑Aggregated Discovery: The Missing Piece for Complete Coverage
A key gap identified in the original MVP design was that while resource requests could be proxied, discovery requests (e.g., /api or /apis) were not similarly forwarded. This meant that even with MVP enabled, a client querying an older server for available APIs might receive an incomplete list, potentially leading to confusion or client‑side errors. In v1.36, this gap is closed with peer‑aggregated discovery. Now, when a discovery request reaches an API server that lacks the latest API groups, the server can proxy the request to a peer that has the complete picture. The result is that both resource and discovery requests are correctly handled, giving clients a consistent view of the cluster's capabilities regardless of which API server they hit during an upgrade.
4. Beta Graduation: What the Default‑Enablement Means for Your Clusters
With v1.36, the Mixed Version Proxy moves to Beta and is enabled by default. This means that any cluster running a multi‑version control plane (common during rolling upgrades) will automatically benefit from MVP's protections. Operators no longer need to manually enable the UnknownVersionInteroperabilityProxy feature gate or worry about complex configuration. However, there are still best practices to follow: ensure your API servers can communicate with each other (the proxy uses standard TLS and the x‑kubernetes‑peer‑proxied header), and be aware that MVP works best when all nodes in the control plane are reachable. For most users, enabling MVP is a straightforward upgrade win that dramatically reduces the risk of upgrade‑related failures.
5. How to Leverage MVP in Your Upgrade Workflow
To get the most out of the Mixed Version Proxy, incorporate it into your standard upgrade procedures. Here are actionable steps:
- Verify connectivity: Ensure all API server instances can reach each other over the cluster's internal network. MVP relies on peer‑to‑peer proxying, so network policies or firewalls should allow traffic between control plane nodes.
- Monitor aggregated discovery: Use
kubectl get apiservicesto confirm that all resources (including CRDs) are properly registered. MVP uses aggregated discovery data to route requests, so an incomplete discovery list can still lead to errors. - Test with canary upgrades: Perform a phased rollout by upgrading one API server at a time and validating that MVP proxies requests correctly. Tools like
kubectl proxyor direct API calls can help verify correct behavior. - Fallback planning: While MVP handles most scenarios, consider enabling
--shutdown‑delay‑durationon API servers to allow in‑flight proxy requests to complete before a server is removed.
By following these guidelines, you can safely accelerate your upgrade cycles and reduce the risk of disruptive false 404s.
Conclusion: A Safer Path to Cluster Upgrades
The graduation of the Mixed Version Proxy to Beta in Kubernetes v1.36 marks a significant step forward for cluster reliability. By transparently routing requests to the correct API server—even during mixed‑version control plane states—MVP eliminates a long‑standing source of operational pain. The shift to Aggregated Discovery and the addition of peer‑aggregated discovery ensure that modern workloads (including CRDs and aggregated APIs) are fully covered. As the feature becomes default‑enabled, Kubernetes operators can upgrade with greater confidence, knowing that false 404 errors are a thing of the past. Embrace MVP in your upgrade workflow and enjoy a smoother, safer Kubernetes experience.