Gartner Predicts 15% of Daily Decisions Will Be Made Autonomously by AI by 2028, But Is the Infrastructure Ready?
2 months ago
Over the past five years, generative AI has taken the market by storm, creating text, dynamic videos, and various other content based on user prompts. Now, Agentic AI promises to bring a fundamental shift for enterprises. Agentic AI not only responds to user prompts but also achieves goals by reasoning, planning, and executing multi-step actions with minimal human intervention.
Gartner predicts that by 2028, at least 15% of daily work decisions will be made autonomously by Agentic AI. Gartner also notes that by the end of 2027, enterprises will cancel over 40% of their Agentic AI projects. Why is Agentic AI so important, yet enterprises struggle to integrate AI into their operations? Gartner believes the biggest obstacles are escalating costs and the complexity of deploying AI agents at scale.
To ensure the successful implementation of Agentic AI, enterprises need to ensure a reasonable ratio of GPUs to CPUs. The "GPU idle tax" is becoming a growing concern as users scale AI, referring to the hidden cost of underutilized GPUs that still incur expenses. Based on data from tens of thousands of clusters, Cast AI reports that GPU utilization is only 5%. GPU idle time is usually not intentional for resource savings but likely an architectural imbalance. In a joint research project, Intel and Georgia Tech found that in agent workloads, tool processing on the CPU accounts for 50% to 90% of total latency; GPUs remain idle while waiting for the CPU to complete work before processing the next batch of tokens, silently increasing operational costs.
As the AI landscape continues to shift and evolve, enterprises need CPUs and GPUs to work together with a reasonable ratio, driving excellent performance and avoiding overpaying for AI infrastructure.
Generative AI vs. Agentic AI
Generative AI and Agentic AI differ fundamentally in how their workflows interact with surrounding systems. Generative AI typically operates in relatively isolated infrastructure. It receives input prompts, processes them using pre-trained models, and returns outputs, usually without deep real-time integration with other systems. Its workflow is largely self-contained, relying on static training data and at most limited retrieval from curated knowledge bases.
In contrast, Agentic AI can act autonomously to achieve goals, requiring continuous communication across multiple components of organizational infrastructure. These systems must be able to orchestrate operations, query real-time databases, interact with APIs, trigger workflows, and dynamically adjust based on feedback from different services. Therefore, Agentic AI relies on an environment where data, tools, and decision layers are interconnected, tightly integrated, and interoperable. Its workflow is inherently more complex and distributed than the relatively isolated processes of generative AI.
Balancing CPU and GPU
Over the past five years of the AI boom, GPUs have been the core of computing power. However, GPUs are not designed to handle the diverse behaviors that define agentic systems, such as retrieving real-time network data, processing continuous sensor streams like cameras and LiDAR for autonomous driving, or navigating complex interfaces like e-commerce platforms to complete tasks. It is in these areas that CPUs play a critical complementary role, providing general-purpose computing, control flow, and system-level coordination that GPUs lack. As Agentic AI evolves, especially in enterprise operational automation, CPU-driven processes are increasingly taking center stage in orchestrating broader workflows, coordinating tools, managing memory and state, and interacting with external systems.
From a workload perspective, this shift brings a more nuanced balance in CPU and GPU utilization. Traditional generative AI workflows spend most of their time on GPUs, with CPUs handling relatively lightweight preprocessing and request management. In contrast, agent workloads, due to frequent tool calls, decision loops, and I/O-intensive operations, may spend most—often the dominant portion—of execution time on CPUs, with GPU usage occurring in short bursts during model inference. This imbalance is particularly pronounced in serving environments, where CPU parallel processing, though flexible, is less scalable than GPUs, meaning tool-heavy agent workloads may saturate CPU resources before GPUs are fully utilized. To maintain high GPU utilization and avoid bottlenecks, Agentic AI systems need high-performance CPUs with sufficient cores and threads to meet task orchestration demands. In modern data centers, properly configuring the number of GPUs and CPUs is crucial; otherwise, facilities lacking sufficient CPU compute power will leave expensive GPU resources idle, reducing system performance and return on investment.
A CPU-Centric Approach
In a paper titled "Toward Understanding, Analyzing, and Optimizing Agentic AI Execution: A CPU-Centric Perspective," Intel researchers, in collaboration with Georgia Tech, describe and analyze system bottlenecks introduced by Agentic AI workloads, emphasizing the central role of CPUs in processing.
To this end, the researchers conducted runtime characterization on different hardware systems, focusing on end-to-end latency, batch throughput, and energy analysis to isolate hardware bottlenecks. The team identified and tested five widely deployed Agentic AI workloads, covering applications such as factual question answering, coding, and scientific tasks, representing diverse model sizes, orchestration patterns, and tool integration strategies. The figure below shows the average percentage of processing time spent on CPU and GPU for these five workloads.
The results show that tool-heavy Agentic AI workloads are severely constrained by tool processing bottlenecks on the CPU, with CPU tasks consuming up to an average of 86% of end-to-end latency. These findings indicate that for most use cases, enterprises should actively implement CPU-centric optimization strategies. However, the specific percentage of CPU vs. GPU processing time varies widely, from Toolformer spending 17% CPU time on math-enhanced tasks to LangChain spending up to 86% CPU time on question-answering tasks. Therefore, considering specific use cases is crucial when determining the most efficient CPU:GPU ratio for an enterprise.
Increasing CPU:GPU Ratio
In the Intel whitepaper "Agentic AI Needs More CPU," the team explores how increasing CPU compute power can alleviate system bottlenecks, thereby maintaining GPU compute saturation to reduce costs. A process built by the team showed that the CPU consumes most of the process time, with the "context supplementation" step taking three times longer than LLM inference time. The researchers also built a second benchmark simulating a developer requesting an AI assistant to write and test code. This benchmark showed that code generation (GPU) completed in 62.8 seconds, while execution on the CPU consumed 64.1 seconds, making the CPU the limiting factor in the process.
According to tests in the whitepaper, researchers found that for current-generation deployments, maintaining a ratio of approximately 0.8:1 to 1.4:1 (CPU to GPU) ensures full GPU utilization. However, as mentioned, this heavily depends on workload and hardware. The figure below shows how model size affects the recommended ratio:
The figure shows that the smaller and more efficient the model, the more CPUs are needed to maintain the same performance. Additionally, the whitepaper notes that as GPU performance improvements continue to outpace CPU improvements, this ratio may need to increase significantly, potentially up to 7:1 in future scenarios depending on workload type and complexity. This research strongly demonstrates that CPUs are central to achieving performance and cost efficiency for next-generation Agentic AI systems.
Intel Xeon Processors, Built for Agentic AI
CPUs handle orchestration, data migration, inference, and system-level coordination, becoming the backbone of the entire AI technology stack. Therefore, choosing the right CPU is crucial for ensuring consistent performance in increasingly complex and hybrid AI environments. Intel Xeon 6 processors combine built-in AI acceleration, infrastructure consistency, and security assurance, providing a robust foundation for modern AI infrastructure.
- Built-in Acceleration: Intel Xeon processors enable enterprises to run inference directly on existing architectures, accelerating time-to-value and reducing reliance on specialized skills, making Xeon stand out. With built-in AI accelerators like Intel AMX and Intel AVX, Xeon delivers strong performance on common inference workloads, achieving up to 50% performance improvement with fewer cores compared to similar competing products. This combination of integrated acceleration, scalability, and operational flexibility makes Xeon a solid foundation for enterprise AI.
- Infrastructure Consistency: Intel Xeon 6 processors are the most widely deployed host CPUs globally. As enterprise AI operations scale and integrate, Xeon ensures infrastructure consistency. These processors support up to 192 PCIe 5.0 lanes, increasing I/O bandwidth, scalability, and architectural flexibility. Like inference workloads, host servers also benefit from Intel Xeon 6 CPUs, with increased memory bandwidth via MCRDIMM and support for CXL 2.0 to expand memory. Industry-leading RAS features help maintain AI cluster stability, while high single-thread performance accelerates tasks like orchestration and scheduling, reducing P99 and tail latency, critical for user-facing AI services.
- AI-Ready Stability and Reliability: Stability and reliability are fundamental for AI applications; enterprises need trusted infrastructure to protect their models and data. Intel technologies are widely recognized for strong product stability and reliability assurance, with approximately 96% of firmware vulnerabilities discovered through proactive internal defenses, compared to only 57% for other similar platforms. Combined with one of the industry's most comprehensive confidential computing portfolios, Xeon 6 is an ideal choice for organizations seeking to build robust, enterprise-grade CPU architectures in heterogeneous AI environments.
In an era where AI workloads are becoming more dynamic, distributed, and tightly integrated with enterprise systems, infrastructure decisions need to balance performance, scalability, and trust. Intel Xeon processors integrate these elements, with built-in acceleration for efficient inference, a consistent and scalable platform supporting growing deployments, and robust protection for critical data and models, meeting the evolving needs of modern AI. By bringing these capabilities together in a widely adopted unified architecture, Xeon enables customers to simplify operations while providing flexibility for scaling AI use cases.
