Google Open-Sources TorchTPU to Loosen Nvidia's Grip

For a decade the argument about AI hardware was settled in software, not silicon. Nvidia's real moat was never the GPU alone but CUDA, and the fact that almost every model written in PyTorch just ran on Nvidia by default. This week that default stopped being automatic.

On Tuesday at PyTorch Conference 2026 in San Jose, Google announced that TorchTPU, its native PyTorch stack for Tensor Processing Units, is now open source on GitHub. Amazon made the same move a day earlier with TorchNeuron for its Trainium chips. Two of the three largest clouds have now published the tooling that lets existing PyTorch code run on their own accelerators with a configuration change instead of a rewrite.

What happened

The Google Cloud post is signed by Mark Lohmeyer, VP and GM of AI and Computing Infrastructure, and Bill Jia, VP of Engineering. The headline claim is structural: "With TorchTPU, TPUs join GPUs and CPUs as a native option for PyTorch." Training, serving and deployment on TPUs now run through the stack developers already use, including vLLM, SGLang, llm-d, TorchTitan, Ray, TorchRec, Hugging Face and LiteRT.

Three pieces of the announcement matter more than the marketing list:

TorchTPU replaces PyTorch/XLA. The old path forced lazy-tensor graph recording, meant adding code to mark compilable regions, and made debugging an exercise in guessing when your code actually executed. TorchTPU uses eager execution by default with PyTorch's one-process-per-device model, then hands production workloads to torch.compile. Google's own docs now recommend starting new workloads on TorchTPU and migrating off PyTorch/XLA.

The serving ecosystem came along. The native vLLM backend on TorchTPU was built with Inferact and ships with custom Pallas kernels, continuous batching, quantization and multi-chip parallelism while preserving upstream worker models, schedulers and API servers. SGLang, llm-d and Ray integrations land in the same release. Google also built dedicated nightly CI pipelines that run the Hugging Face Transformers suite on TPU infrastructure, which is the unglamorous part that decides whether a backend is trustworthy in production.

LiteRT joined the PyTorch ecosystem. LiteRT, formerly TensorFlow Lite, now converts PyTorch models via litert-torch.convert() and runs them on Android, iOS, macOS, Windows, Linux, web and IoT hardware, with acceleration across CPUs, GPUs and NPUs.

Amazon's move one day earlier was deliberately similar: TorchNeuron lets PyTorch apps run on Trainium by changing a single line of configuration, and AWS said it is contributing upstream code to PyTorch that decouples framework features from specific chips.

Why the CUDA moat was always software

PyTorch came out of Meta in 2016, years before anyone was arguing about inference margins. Nvidia's GPUs were already the standard research hardware, so a large share of PyTorch's internals were tuned for them, and the framework's gravity pulled every new model, library and tutorial toward the same vendor. Meta handed PyTorch to the Linux Foundation in 2022, but the center of gravity did not move.

That is why Google and Amazon are attacking at the framework layer instead of the chip layer. A faster accelerator is worth very little to a team whose training script, serving stack and observability tooling are all bound to CUDA. What matters is whether the migration is a config change or a quarter of engineering time.

Reuters reported in December 2025 that Google was working with Meta to erode Nvidia's software advantage through exactly this route. TorchTPU going open source is that effort shipping nine months later, and the PyTorch blog frames it as "an open source collaboration between Google and Meta."

The advantage this creates for Google is not that TPUs become better than GPUs. It is that TPUs become an option a team can price against. Once the switching cost drops from a rewrite to a pip install torch_tpu plus a config change, procurement conversations change shape, and so does the negotiating position of every team renting capacity.

What migration actually looks like

Installing is a single command against the PyTorch CPU index, a hint that you should not pull the multi-gigabyte CUDA build onto a TPU VM. In Hugging Face's Transformers v5, deploying to TPUs is device_map="tpu". Diffusers runs natively with eager mode, torch.compile and tensor parallelism. Distributed training goes through torch.distributed with tpu_dist, and TorchTitan on TPU is published as a fork at google-pytorch/torchtitan for teams that want a reference recipe rather than a blank page.

The caveats are worth stating plainly. Open-source tooling does not make capacity appear, TPU quotas still gate real workloads, and the hard problems in a migration are rarely the device backend. They are the custom kernels someone wrote for a specific GPU generation, the profiling workflows tuned to Nsight, and the twelve places where a library quietly assumed cuda. Google's own answer to part of that is the TPU agent suite, an early-preview set of plugins for coding harnesses that automates onboarding, diagnostics and inference performance tuning.

There is also the older truth that portability cuts both ways. The same open backend that lets a PyTorch shop try TPUs also makes it trivial to walk back to GPUs, or to try Trainium, or to move to whichever provider offers a better rate next quarter. Openness here is a competitive weapon aimed at the incumbent, and it will not stay pointed in one direction.

The new economics for agent builders

The timing matters more than the technology. As reasoning and agentic models scale, inference cost and latency become the primary production bottleneck. An agent that calls a model forty times to complete one task converts every token-price difference into a margin difference, and every point of throughput per dollar into the ability to run one more customer.

That is exactly the workload Google is targeting. Lohmeyer and Jia write that TorchTPU is integrated "into widely used open-source serving engines and paired them with Kubernetes-native fleet routing across TPUs and GPUs to help you optimize throughput per dollar." For teams building on vLLM or SGLang, the realistic near-term win is not switching vendors but gaining a credible second quote and the ability to route batch work to whichever accelerator is cheapest that week.

The second-order effect is on the framework itself. Amazon is pushing upstream changes that make PyTorch hardware-agnostic by default, and Google is shipping a native backend built with Meta's PyTorch team. Two clouds are now funding the same goal: a PyTorch that treats the chip as an interchangeable detail.

What to watch

Whether the upstream contributions land. Amazon's commit to decouple PyTorch features from specific chips is the most consequential line in either announcement, because it changes the default for everyone rather than the experience for one vendor's customers.

Whether Anthropic and OpenAI start publishing TPU-optimized serving recipes. Neither has an interest in making Nvidia's pricing power permanent, and both run inference at a scale where a percentage point of throughput is real money.

And whether Nvidia's answer is software or price. Every previous attempt to break CUDA failed on ecosystem inertia, not on benchmark numbers. This time the attack is aimed directly at the inertia, from two companies that also happen to sell the alternative.