Early inference tests show major efficiency and latency gains, but production scale and realistic agent workloads remain important tests.
Last updated on
AI Summary
OpenAI’s first custom inference chip, Jalapeño, beat Nvidia Blackwell systems in early tests, delivering higher efficiency and substantially lower latency across three large language models.
Using InferenceX on GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T, OpenAI reported 1.5 to 1.9 times greater peak work per watt and 1.7 to 3.6 times lower end-to-end latency than Nvidia GB200 or GB300 systems. Jalapeño uses HBM4 memory and was designed by OpenAI, with Broadcom handling silicon implementation and networking and Celestica supporting system integration. SemiAnalysis observed lab runs and said the chip also surpassed every tested AMD and Google accelerator on several open models, though it did not independently execute the full benchmark suite.
The tests used fixed 8K-input, 1K-output requests, not realistic AgentX workloads. OpenAI plans initial infrastructure deployment by the end of 2026 and broader production during 2027, when manufacturing scale, software maturity, and rack reliability will determine whether the lead persists.
OpenAI has published the first performance results for Jalapeño, its custom AI inference chip, and the initial numbers are unusually strong for first-generation silicon. Across three large language models, the company reports 1.5 to 1.9 times more work per watt at peak throughput and 1.7 to 3.6 times lower end-to-end latency than the Nvidia systems used for comparison.
The broader claim is even more striking. SemiAnalysis, whose researchers observed benchmark runs in OpenAI’s lab, says Jalapeño beat every Nvidia, AMD, and Google chip it had tested on several open models. That makes Jalapeño more than an experiment in reducing OpenAI’s dependence on third-party chips. It is a credible high-performance inference platform.
The distinction between training and inference matters. Training creates an LLM, while inference runs the finished model each time someone sends a ChatGPT prompt, launches a Codex task, or calls an API. A more efficient inference system can improve response speed, increase capacity, and reduce the cost of serving every request.
OpenAI tested Jalapeño with the public InferenceX benchmark on GPT-OSS 120B, DeepSeek R1 670B, and Kimi K2.5 1T. The benchmark measures complete serving systems rather than relying only on theoretical chip specifications.
Jalapeño reached the benchmark’s Pareto frontier, meaning no compared system delivered both greater throughput and lower latency at the same operating point. That is significant because inference platforms normally improve aggregate throughput by batching more requests, which can make each individual user wait longer. Jalapeño’s advantage appeared at high-throughput and highly interactive settings.
OpenAI normalized the results using published package power ratings: 700 watts for Jalapeño, 1,200 watts for GB200, and 1,400 watts for GB300. It says Jalapeño’s measured sustained consumption remained at or below 550 watts during the tested workloads, although the efficiency calculations still used the higher 700-watt rating.
Some charts show much larger gains, including more than 100 times the throughput at a previous Nvidia token-rate target. Those figures compare specific points on the latency-throughput curve, not average performance across every workload. The 1.5-to-1.9-times peak efficiency gains are the more useful headline numbers.
Jalapeño Is an Inference System, Not Just a Chip
Jalapeño’s performance comes from designing the accelerator, memory, networking, software, and rack-scale system together. OpenAI architected the processor, while Broadcom handled silicon implementation and networking technology. Celestica contributed board, rack, and system integration expertise.
An LLM request has different computational phases. Prefill processes the input prompt and tends to require substantial arithmetic performance. Decode generates the answer token by token and depends more heavily on memory bandwidth. Moving model state between processors can introduce another bottleneck, especially when the key-value cache holding the conversation’s working state must cross a network.
OpenAI designed Jalapeño to keep model state local and reduce unnecessary data movement. Its connected system can allocate compute, memory, and networking resources according to the current phase without dividing the fleet permanently into separate prefill and decode pools. That flexibility should be useful for agent workloads, where prompt lengths, cached context, concurrency, and output requirements can change rapidly.
SemiAnalysis also reports that Jalapeño uses HBM4 memory and achieved its current results without speculative decoding or multi-token prediction. Those techniques can increase performance by proposing or producing several tokens in one decoding step, so strong single-token-prediction results leave OpenAI with additional optimization options.
The Nvidia, AMD, and Google Comparisons Need Context
The claim that Jalapeño beats Nvidia, AMD, and Google comes from SemiAnalysis’s testing, not solely from OpenAI’s public appendix. OpenAI’s detailed tables identify Nvidia GB200 and GB300 as the comparison systems. The broader conclusion, repeated in TradingView’s report, covers the accelerators and configurations SemiAnalysis had been able to test.
That conclusion should not be interpreted as a universal win on every model or production workload. SemiAnalysis says OpenAI supplied the numbers and that its researchers verified benchmark runs in the lab, but they did not execute the entire InferenceX suite independently. The published tests used a fixed 8K-input, 1K-output request shape rather than the newer AgentX workload, which measures long-context, multi-turn sessions with cache reuse, tool pauses, routing, and parallel subagents.
The generation comparison also deserves attention. Blackwell is established production hardware, while Jalapeño uses newer memory technology and remains in the engineering and qualification phase. Nvidia’s Vera Rubin platform, which also uses HBM4 and is now ramping at cloud partners, is a more forward-looking competitor. SemiAnalysis says Jalapeño still leads the available Rubin figures on throughput per megawatt, but both platforms have immature software and could improve substantially.
Jalapeño’s early lead is therefore meaningful, but it is not the final result of a settled hardware race.
AI Helped Design and Program the Chip
OpenAI and Broadcom moved Jalapeño from initial design to manufacturing tapeout in nine months. OpenAI says its models helped engineers explore implementations, optimize arithmetic circuits, shorten verification cycles, and iterate against real LLM workloads.
AI also contributed to the software stack. Using Codex with GPT-Astra, the team brought three open-weight models to high performance in two months. On selected GPT-OSS attention and mixture-of-experts blocks, AI-generated implementations ran 1.5 to 1.8 times faster than existing human-written versions. OpenAI correctly limits that claim to individual blocks rather than presenting it as a full-model speedup.
This development loop could become as important as the silicon itself. Nvidia’s CUDA advantage rests partly on years of optimized libraries, tools, and developer support. AI-generated kernels do not erase that ecosystem, but they may help custom-chip teams reach competitive software performance much faster than previous accelerator projects.
Faster AI Products Are the Immediate Goal
For ChatGPT users, Jalapeño could reduce the delay before an answer appears and increase token-generation speed once it starts. For Codex and other agents, lower latency can have a larger cumulative effect because one task may require many sequential model calls. Saving a fraction of a second at each step can shorten the complete workflow.
Higher throughput per watt also lets OpenAI serve more requests within a fixed data-center power budget. That can improve availability during periods of heavy demand and lower the infrastructure cost of each successful task. OpenAI has not announced corresponding ChatGPT or API price cuts, however, so lower serving costs should not automatically be treated as lower customer prices.
Jalapeño is also a source of negotiating and operational leverage. OpenAI gains another supply option and greater control over its inference economics, but it is not abandoning merchant accelerators. The company explicitly says it will continue deploying Nvidia hardware and chips from other partners for training and inference.
Deployment Will Be the Harder Test
OpenAI plans to begin installing Jalapeño in its compute infrastructure by the end of 2026. Before then, the company must complete production qualification, mature the software stack, validate additional models, and prove that the system can operate reliably at data-center scale. Broader production is expected to ramp during 2027.
The roadmap already extends beyond the first chip. OpenAI says Jalapeño Gen 2 is deep in development and Gen 3 is taking shape. Future versions can incorporate production data, new LLM architectures, improved decoding methods, and lessons from deploying the first-generation system.
Final Thoughts
Jalapeño’s most important achievement is not one benchmark victory over Blackwell. It is that OpenAI’s first custom AI chip has already reached a competitive latency-throughput frontier while running large models from several developers.
The unanswered questions now concern production rather than potential. OpenAI still needs to reproduce these gains on long-context agent workloads, manufacture enough chips, and operate complete racks reliably. If Jalapeño maintains its efficiency advantage after that transition, OpenAI will have built more than an Nvidia alternative. It will have created a faster feedback loop connecting its models, software, products, and physical infrastructure.
Frequently Asked Questions
4 questions
1
What Is OpenAI’s Jalapeño AI Chip?
Jalapeño is OpenAI’s first custom processor designed specifically for LLM inference, the process of running trained models and generating responses. OpenAI designed its architecture, Broadcom handled silicon implementation and networking, and Celestica contributed system integration. It is intended for OpenAI’s internal infrastructure rather than sale as a general-purpose consumer or data-center chip.
2
Does Jalapeño Outperform Nvidia, AMD, and Google Chips?
Jalapeño outperformed the tested Nvidia systems in OpenAI’s published InferenceX results, delivering 1.5 to 1.9 times higher peak work per watt and substantially lower latency. SemiAnalysis says it also beat every AMD and Google accelerator the firm had tested on several open models. The result does not cover every possible model, configuration, or production workload.
3
When Will OpenAI Deploy Jalapeño?
OpenAI plans to begin deploying Jalapeño within its compute infrastructure by the end of 2026. The first installations will follow continued production qualification, software development, model validation, and operational testing. SemiAnalysis expects manufacturing volume to increase through 2027, so the chip’s effect on overall OpenAI capacity will probably be gradual rather than immediate.
4
How Could Jalapeño Improve ChatGPT and Codex?
Jalapeño could make ChatGPT answers arrive faster, increase the responsiveness of Codex sessions, and give OpenAI more capacity during periods of high demand. Its efficiency is particularly valuable for agents because their tasks often involve many model calls in sequence. OpenAI has not promised specific price reductions or confirmed when individual products will begin using the chip.
