The artificial intelligence boom initially centered on GPUs and memory, but the rise of agentic AI is creating unexpected demand for CPUs. Amazon Web Services has reportedly experienced longer waits for CPU server capacity, prompting engineers to conserve CPU cycles as AI workloads strain its cloud infrastructure, tells IEEE Spectrum.
Traditional AI inference relies heavily on GPUs because their parallel-processing capabilities are well suited to the mathematical operations behind large language models. Agentic AI changes the computing workload. Agents can autonomously use software, access files, make API calls, browse the internet, and create sub-agents. While model inference generally remains on GPUs, many supporting operations run on CPUs.
These tasks include interpreting model output, determining which tools to use, executing code, collecting results, and returning information to the model. AMD says seven of eight stages in realistic agentic AI pipelines can run entirely on CPUs. Researchers have also found that CPUs and GPUs frequently wait for each other, creating opportunities for scheduling improvements that could reduce overall latency.
Tokenization adds another bottleneck. Every time an agent makes a tool call, returned information may need to be tokenized along with an increasingly long context. As sequences reach hundreds of thousands of tokens, CPUs with insufficient cores can delay instructions sent to GPUs. Research indicates that increasing CPU core counts can reduce time-to-first-token latency by roughly 1.5 to seven times for long sequences.
Safety checks can further increase CPU workloads because many guardrails involve rule-based processing or small models that run efficiently on CPUs.
Hardware manufacturers are responding. Intel has sold out server CPUs through at least year-end, AMD has increased its server CPU forecast, and Arm, Qualcomm, and Nvidia are developing processors aimed at agentic workloads. The shift suggests CPUs are becoming essential to AI infrastructure, potentially creating shortages and higher prices across server and consumer markets.
