Google accompanied the recent launch of its Gemini AI models with the latest version of its flagship tensor processing unit (TPU) for AI training and inference, in what appears to be an attempt to usurp Nvidia's own market-leading GPUs.
TPU v5p – Google's most powerful custom-designed AI accelerator – has been deployed to power the company's 'AI hypercomputer'. This is a supercomputer architecture built specifically to run AI applications, rather than supercomputers that normally run scientific workloads, as TPUs are not suited for this.
The latest version of the TPU has 8,960 chips per pod (making up the system), up from 4,096 in v4, and is four times more scalable in terms of total availability of FLOPs per pod. These new pods offer a throughput of 4,800 Gbps. The new pods also have 95 GB of high-bandwidth memory (HBM) versus 32 GB of HBM RAM in TPU v4.
Nvidia H100 vs Google TPU v5p: Which is Faster?
Unlike Nvidia, which offers its GPUs to other companies, Google's custom TPUs remain internal for use in its own products and services. Google's TPUs have long been used to power its services including Gmail, YouTube and Android, and the latest version is also used to train Gemini.
Google's v5p TPUs are up to 2.8x faster at training large language models than TPU v4, and offer 2.1x value for money. Although the intermediate version, TPU v5e, released earlier this year, offers the best value of all three, it is only 1.9 times faster than TPU v4, making TPU v5p the most powerful.
It's even powerful enough to rival Nvidia's highly sought-after H100 GPU, one of the best graphics cards available for AI workloads. He claims this component is four times faster at training workloads than Nvidia's A100 GPU the company's own data.
Google's TPU v4, meanwhile, is estimated to be between 1.2 and 1.7 times faster than the A100, according to Google. research published in April. Incredibly rough calculations suggest that the TPU v5p is therefore roughly between 3.4 and 4.8 times faster than the A100 – making it comparable or even superior to the H100, although more detailed benchmarking is needed before any conclusions can be drawn.