Everyone wants their models running on the device. On-device inference is private by default, it has no network latency, and it does not bill you per token. The hard part is the gap between a model that trains well in a notebook and a model that actually runs well on a specific phone, NPU, or microcontroller. That gap is where most edge AI projects quietly stall.
The gap exists because deployment is not one decision. It is a coupled search across the model architecture, the quantization scheme, the compiler schedule, the runtime, the backend, and the target hardware, and the best choice at each layer depends on every other layer. The recipe that is optimal for a Blackwell GPU is wrong for a Cortex-M microcontroller. Doing this by hand, per model, per chip, does not scale.
The interesting shift over the last year is that this search is being handed to agents. Instead of an engineer hand-tuning each layer, you describe the goal, run this model under a latency and accuracy budget on this hardware, and an agent composes the recipe, runs it, measures the result, and iterates. Four capabilities make that practical: goal-driven optimization, an integrated optimization workflow, a deployment-ready model zoo, and portable deployment across frameworks and chips. This is a field guide to all four.
Goal-driven optimization: describe the outcome, let the agent find the recipe
The oldest form of this idea is the autotuning compiler. Apache TVM has shipped three generations of it: AutoTVM, the Ansor auto-scheduler, and MetaSchedule. Rather than hand-writing kernels, you give the compiler a tensor expression and it searches the space of loop tilings, vectorizations, and thread bindings, guided by a learned cost model and measured on the real target hardware. This exists because the per-operator schedule space is already too large to tune by hand, before you even reach the rest of the stack.
Hardware-aware neural architecture search pushes the same idea up a level, treating the model architecture, the quantization policy, and the compiler schedule as one multi-objective problem rather than separate choices. Work like JAQ in 2025 searches architecture and low-bit quantization together, using Pareto methods and surrogate predictors so it does not have to measure every candidate on-device.
The 2025 to 2026 wave is agents that write and optimize the low-level code directly. Stanford's KernelBench, released in early 2025, is the standard evaluation: 250 tasks across four difficulty levels, scored by a metric that only counts a generated kernel if it is both correct and faster than PyTorch. Sakana AI's robust-kbench added a harder harness that tests forward and backward passes with randomized initialization to stop models from gaming the reward. And DeepMind's AlphaEvolve, announced in May 2025, is the clearest proof that this works at scale. It pairs a Gemini model that proposes program variants with an automated evaluator that scores them, keeping what wins. In production it found a 23 percent speedup on a matmul kernel that cut Gemini's own training time by about 1 percent, delivered up to a 32.5 percent speedup on a FlashAttention kernel implementation, and recovered on average 0.7 percent of Google's worldwide compute through a scheduling heuristic that ran in production for more than a year. It also found a way to multiply two 4 by 4 complex-valued matrices in 48 scalar multiplications, one fewer than the best method known since 1969.
The same pattern is now being applied to the deployment recipe itself: agents that select quantization configurations from a goal and a hardware target, building on a decade of mixed-precision search methods. The through-line is simple. You state the outcome you want and the constraints you have, and the agent runs a compile, verify, and profile loop until it finds a recipe that hits them. The judgment that used to require a compiler specialist becomes a loop.
The integrated workflow: convert, optimize, quantize, profile, validate
Once you know the recipe, running it is a pipeline, and each stage has matured a lot recently.
Convert. The model has to leave its training framework in a form the target runtime can execute. In PyTorch this now goes through torch.export and the dynamo-based ONNX exporter, the recommended path since PyTorch 2.5, which captures the whole graph ahead of time. That same capture underpins ExecuTorch, PyTorch's on-device runtime, which reached its 1.0 release in October 2025. On the other side, TensorFlow Lite was renamed LiteRT in September 2024 and now runs models exported from PyTorch and JAX as well, a signal that the on-device runtime is meant to be framework-agnostic.
Graph optimize. Before any numerics change, the compiler rewrites the graph: fusing operators, folding constants, and transforming memory layouts so the hardware does less work. ONNX Runtime exposes this as tiered levels, from basic constant folding up to full layout optimization. TensorRT goes further, fusing layers and then auto-tuning kernels for the specific GPU before serializing a hardware-specific engine. OpenVINO and TVM do their own device-specific fusions. The theme is that a graph is not a fixed thing; it is rewritten for the target.
Quantize. This is where most of the speed and memory savings come from, and it is the stage that moved fastest. The reliable weight-only methods, GPTQ and AWQ, shrink models to 4-bit while keeping compute in higher precision; SmoothQuant enables 8-bit weights and activations so you can actually use integer compute units. The bigger story is the arrival of 4-bit floating point. NVIDIA's NVFP4 format packs weights into 16-value blocks with a shared scale and reports roughly 3.5 times less memory than FP16 with under 1 percent accuracy loss on models like DeepSeek-R1. OpenAI shipped its open gpt-oss models natively in the related MXFP4 format, which is what lets the 120 billion parameter model fit on a single 80 GB GPU and the 20 billion parameter model fit in 16 GB. Quantizing the key-value cache in FP8 buys back roughly half the memory a long context would otherwise consume.
Profile and validate. This is the stage teams skip and regret. You have to measure latency on the actual target device, not estimate it from FLOPs, because a TensorRT engine tuned for one GPU architecture is neither portable to nor optimal on another. And you have to re-check accuracy after every step, because aggregate benchmarks hide the regressions that matter. A vivid example: naive FP8 key-value cache quantization can look fine on average while a long-context retrieval task collapses from 91 percent accuracy to 13 percent, recoverable only with a smarter accumulation strategy. That is why the workflow is a loop, quantize then check, not a straight line.
A deployment-ready model zoo: models that actually run on the device
A model zoo for the edge is not a list of models you could download. It is a list of models that, once optimized, actually run on the target at acceptable quality. The categories worth knowing in 2025 to 2026:
Language. Small language models crossed the line into genuine phone-class usefulness. Llama 3.2 at 1 and 3 billion parameters, Phi-4-mini, Gemma 3 at 1 billion and the multimodal Gemma 3n, Qwen3 from 0.6 to 4 billion, and the SmolLM family all run on-device. Reported throughput is real: a 4-bit Llama 3.2 1B decodes around 50 tokens per second on a recent phone's CPU through ExecuTorch, and Gemma 3 1B in 4-bit reaches thousands of tokens per second of prefill on a flagship Android GPU. One honest caveat: on-device numbers are mostly vendor-claimed, and independent measurements sometimes land well below them, so treat a single tokens-per-second figure as a starting point, not a guarantee.
Vision-language. SmolVLM at 256 and 500 million parameters runs a single image in under a gigabyte of memory; Apple's FastVLM, Moondream, PaliGemma 2, and MiniCPM-V round out the range, with MiniCPM-V demonstrated generating around 18 tokens per second on an iPhone.
Vision-language-action. This is the robotics frontier, and it is where the real-time bar bites. OpenVLA is a 7 billion parameter model that runs at roughly 6 hertz on a desktop GPU; Physical Intelligence's pi0 uses action chunking to drive control at up to 50 hertz; Google's Gemini Robotics On-Device and NVIDIA's GR00T aim the same capability at on-robot hardware. Manipulation wants tens of hertz, which is exactly why these models lean on chunked or diffusion-based action heads rather than generating one action at a time.
Speech. Speech is the most edge-ready category. For recognition, Whisper and its distilled and streaming descendants like Moonshine run near real-time on a Raspberry Pi. For synthesis, the 82 million parameter Kokoro model, Piper, and even a 15 million parameter TTS model synthesize speech on a CPU with no accelerator.
Vision. Efficient vision backbones, MobileViT, FastViT, EfficientViT, and the small YOLO detectors, deliver millisecond inference on mobile chips. One deployment fact worth internalizing: transformers are markedly slower than convolutional networks on today's NPUs. On Qualcomm's own measured hardware, a quantized YOLOv8n runs in well under a millisecond while a ViT classifier takes several, which often decides the architecture choice for you.
Flexible deployment: any framework, any chip
The last capability is portability, and it is the one with the most fine print. The promise is train anywhere, run anywhere, usually through ONNX as the interchange format: export from PyTorch, TensorFlow, or JAX, then run through a common runtime. The promise mostly holds, and it erodes at two predictable seams. The first is operator and opset coverage, where a backend does not implement an operator or supports it only at a certain version. The second is quantization portability: 8-bit integer quantization is well standardized, but FP8, sub-4-bit, and per-channel schemes fragment across runtimes, and the same model can even produce slightly different numbers on different runtimes because their kernels differ.
Underneath the interchange format is a fragmented hardware reality. Each target has its own optimal stack: TensorRT for NVIDIA GPUs, OpenVINO for Intel CPUs, GPUs, and NPUs, Qualcomm's QNN for Hexagon, Core ML for Apple silicon, and Arm's Ethos-U path for microcontrollers. Runtimes paper over this with pluggable backends: ONNX Runtime calls them execution providers, ExecuTorch calls them delegates, and Windows ML, which reached general availability in September 2025, manages vendor providers for you. The point of all of them is to let one exported model retarget CPU, GPU, NPU, or MCU without a rewrite.
The hardware itself is why this matters now. The NPU became a standard part. Microsoft's Copilot+ PC bar is a neural processing unit rated at 40 or more trillion operations per second, met by Qualcomm's Snapdragon X Elite at 45, Intel's Lunar Lake NPU at around 48, and AMD's Ryzen AI 300 at around 50; Apple's M4 Neural Engine is rated at 38. At the other extreme sits the microcontroller, where the binding constraint is kilobytes of memory, not throughput. LiteRT for Microcontrollers has a core runtime that fits in roughly 16 kilobytes on an Arm Cortex-M3, and Arm's Ethos-U microNPUs bring a few trillion operations per second to that class of device. A model that has to run across that whole range, from a Blackwell GPU to a Cortex-M, is exactly the kind of target no single hand-tuned recipe can serve.
The loop is the product
Step back and the four capabilities are one thing. Goal-driven optimization decides what recipe to try. The integrated workflow runs it. The model zoo supplies a starting point that is known to be deployable. Portable deployment lands it on the target. And the reason to tie them together with an agent rather than a checklist is that the search space is too large and too coupled for a person to cover by hand, model by model and chip by chip, while the cost of a wrong choice, an operator the backend cannot run, a quantization scheme that quietly destroys long-context accuracy, is high and easy to miss.
What makes this trustworthy is not the agent's cleverness. It is that every step ends in a measurement: real latency on the real device, and accuracy re-checked against the baseline after every transformation. An agent that optimizes against a benchmark it cannot verify will happily ship you a model that is fast and wrong. The teams that get this right treat the agent as the thing that explores the enormous space, and the profiler and the eval set as the thing that keeps it honest. Describe the goal, let the agent build the recipe, and then make it prove the result on the hardware you actually ship.
