Home · Chip or Software? · MAX on Qualcomm

Was it the chip or the software?

Started September 2026 · in progress

A 2025 UCSD study (arXiv:2507.00418) served 12 LLMs on the Qualcomm Cloud AI 100 Ultra and on 4×/8× NVIDIA A100, using vLLM on both. It found competitive energy efficiency, a lead in tokens per watt on 3 of 12 models, and much finer hardware allocation: a small model on one 36 W device, a 70B model on one card. Its tables also show headroom that looks like software: similar models needing very different amounts of silicon, and batching capped at 4.

Since then, Qualcomm acquired Modular, and ModCon 2026 showed MAX running on Cloud AI 100 Ultra. This project re-runs the study with MAX as the serving engine to measure how much better the chip gets with a newer software stack.

Hypotheses, stated before any runs:

H1: on Qualcomm, MAX substantially raises throughput over the paper's numbers at similar power
H2: on the A100, MAX matches or beats vLLM in throughput and tok/s/W
H3: a right-sized A100 (1 GPU for ≤8B models) raises the A100's tokens per watt
H4: one Mojo kernel runs unchanged on Apple, NVIDIA, and Qualcomm silicon

Code & data on GitHub · About this work

Posts

Chip or software? Why I'm re-running the Qualcomm Cloud AI 100 vs A100 study on Modular MAX

28 Sep 2026

A 2025 UCSD study served 12 LLMs on Qualcomm's Cloud AI 100 Ultra and on NVIDIA A100s, both through vLLM. It found competitive energy efficiency, a clear lead on some models, and far finer hardware allocation. Its own tables also show headroom that looks like software. Qualcomm now owns Modular, so I'm re-running the study on MAX to measure how much of that headroom a new stack can claim.

Testing the stopwatch: building an LLM benchmark on a MacBook before renting GPUs

28 Sep 2026

Before re-running a Qualcomm vs NVIDIA LLM-serving study on rented GPUs, I built and tested the benchmark on a MacBook. The pilot caught a power-sampling bias that inflated efficiency by up to 12× and a GPU wake-up delay hiding in the latency numbers, and my client matched Modular's own benchmark tool within 1–3%.