Tech Insights

Google Cloud’s Open AI-Native Stack Turns Kubernetes into an AI Control Plane

Google Cloud is assembling an open-source AI stack around Kubernetes, linking distributed inference, reinforcement learning, Ray workloads, agent orchestration, sandboxing, and accelerator backends.

Featured image for Google Cloud's Open AI-Native Stack Turns Kubernetes into an AI Control Plane

Kubernetes became the default control plane for many web services because it gives teams one way to schedule containers, manage resources, and operate workloads across a cluster. AI workloads have exposed the limits of treating that problem as ordinary container orchestration. Large models need specialized accelerators, inference needs memory-aware routing, agents may execute untrusted code, and training and serving often compete for the same expensive hardware.

At KubeCon + CloudNativeCon North America 2026, Google Cloud described an open AI-native stack that brings those concerns into a Kubernetes-centered architecture. In its official overview, Google Cloud listed OpenRL, llm-d 1.0, Ray on GKE with KubeRay, Agent Substrate, gVisor, and TorchTPU as parts of the stack.

The announcement is not one product replacing Kubernetes. It is an architectural direction: use open projects and Kubernetes-native interfaces to manage the different stages of an AI system, from post-training and inference to agent execution and accelerator access.

Why AI workloads need more than a generic cluster

A conventional service cluster usually schedules containers and keeps them available. An AI platform has to answer additional questions. Which accelerator has the right memory and interconnect? Can a request reuse a prefix cache? Should prefill and decode run on different resources? How should an agent pause, resume, and retain state? What happens when model-generated code needs to run in a shared environment?

Teams can build separate systems for each question, but that often creates the fragmentation Google Cloud is trying to address. A Ray or Slurm environment may sit beside Kubernetes. An inference stack may use its own scheduler. Agent workloads may need another execution service. Each system can be useful on its own, yet the result is duplicated operations, fragmented capacity, and multiple ways to express policy.

The open AI-native stack treats these as connected layers. Kubernetes provides the operating surface, while specialized projects provide the behavior that generic scheduling does not understand.

Distributed inference becomes a platform concern

The stack includes llm-d 1.0, described by Google Cloud as an open distributed inference control plane. Its focus is not simply starting a model server. It includes prefix-cache-aware routing, disaggregated serving, and multi-tier key-value cache management.

That matters for long-context and high-concurrency workloads. A request may benefit from being routed to a worker that already holds part of the relevant context. Prefill and decode can be separated when they have different resource profiles. Model state can be moved across memory tiers instead of assuming every token must remain in the accelerator’s fastest memory.

For platform teams, the benefit is a more explicit place to express inference behavior. The trade-off is complexity. Cache-aware routing and disaggregated serving can improve utilization, but they also require careful observability, failure handling, and workload-specific testing. A control plane that understands model state still needs to integrate with the application’s latency and quality requirements.

Training and reinforcement learning share the same fabric

OpenRL is presented as a Kubernetes-native post-training service. Reinforcement learning pipelines often require trainers, rollout workers, data movement, and accelerator scheduling to stay coordinated. If those pieces are managed in isolation, resources can sit idle while one stage waits for another.

Putting post-training into the same broader orchestration model can make resource ownership and scheduling easier to reason about. Teams can use Kubernetes concepts for placement, capacity, and lifecycle management while still using an AI-specific service for the details of reinforcement learning.

That does not make a reinforcement-learning pipeline simple. Training data, evaluation, checkpointing, and reward design remain separate engineering problems. The architectural value is that the lifecycle of those jobs can become visible to the same platform team that manages inference and other workloads.

Ray remains useful, but less isolated

Google Cloud also highlights Ray on GKE with KubeRay. Ray is used for distributed Python workloads, data processing, training, and inference. KubeRay provides a way to manage Ray clusters through Kubernetes, while Kueue can help schedule workloads against shared capacity.

This is a pragmatic part of the design. Teams do not need to discard a framework simply because they adopt Kubernetes. Instead, the platform can manage the lifecycle of Ray workloads and place them alongside other AI services. That can reduce the need for a separate cluster with a separate operator experience.

The trade-off is that a framework inside Kubernetes still has its own concepts and failure modes. Operators need to understand both layers: Kubernetes scheduling and Ray’s distributed execution model. The integration is valuable when it clarifies ownership and capacity, but it does not remove the need for expertise in the underlying framework.

Agents make isolation part of the architecture

Agent workloads change the risk profile of a cluster because an agent may generate shell commands, scripts, or other tool actions. In a multi-tenant environment, the execution boundary becomes as important as scheduling and scaling.

Google Cloud includes Agent Substrate as an orchestration runtime for multi-turn AI agent workflows and gVisor as a sandbox for stronger system-call isolation. Together, these components address two related questions: how an agent’s lifecycle is managed, and how its code is contained when it runs.

That separation is useful. An orchestration layer can manage state, suspension, resumption, and execution targets, while a sandbox provides an isolation boundary around the workload. Neither layer should be treated as a complete security guarantee. Teams still need least-privilege access, network controls, image and dependency hygiene, auditability, and clear limits on what an agent is allowed to do.

Accelerator access becomes less opaque

TorchTPU is another part of the stack, bringing native PyTorch support across Cloud TPUs and GPUs inside upstream inference tools such as vLLM and SGLang. The broader goal is to reduce the amount of platform-specific work needed to move from a model written in a familiar framework to a production serving environment.

This is important because accelerator choice often creates a software boundary. A team may have a working model in PyTorch but still face changes in kernels, runtimes, deployment manifests, and serving integrations before it can use a different class of hardware. A more open backend path can make that transition easier to evaluate.

Portability still has limits. Hardware-specific performance, memory layout, compiler behavior, and model support can vary. “Runs on” is not the same as “runs with the same cost, latency, or operational profile,” so teams should measure their own workloads rather than assuming a backend removes every difference.

What the architecture changes for platform teams

The main change is an ownership model. Instead of treating AI as a collection of isolated projects, a platform team can provide common interfaces for scheduling, serving, sandboxing, and accelerator access. Researchers can focus on models and training jobs, application teams can focus on agent behavior, and operators can maintain the shared runtime and policies.

The cost is architectural discipline. A modular stack creates choices about which components are managed centrally, which are exposed to teams, and how versions are coordinated. It also creates a need for end-to-end observability across Kubernetes, the inference control plane, the accelerator runtime, and the agent sandbox.

Teams should therefore adopt the stack by workflow rather than by component count. Start with one workload, define the handoff between model serving and orchestration, specify the security boundary, and measure what the platform reduces. A stack is useful when it removes repeated work or improves reliability; it is not useful merely because it contains more projects.

The bigger takeaway

Google Cloud’s announcement reflects a broader shift in how AI infrastructure is being built. Kubernetes is no longer only the place where a model server happens to run. It is becoming a place where teams can coordinate model state, distributed jobs, agents, accelerators, and isolation boundaries through one operating model.

The open-source pieces matter because no single project owns every layer of the AI lifecycle. The challenge is making those pieces work together without recreating the fragmentation they were meant to replace. For platform teams, the opportunity is a shared AI control plane. For operators, the work is proving that the abstractions remain understandable when the workloads become expensive, stateful, and capable of taking action.

This article is a general information summary based on the cited public sources. It is not legal, cybersecurity, or compliance advice.

Written By

PragnaTeja Bandiboyina

Web Designer

PragnaTeja Bandiboyina shares practical publishing insights, comparisons, and WordPress-focused growth guidance for modern web teams.

207 articles published Member since 2026