Axon Labs
All insights

From the engineering desk

Insight11 min read20 July 2026

Can Your Product Run a Language Model? SLMs on Embedded Hardware

Small language models fit in 1–2 GB and run without the cloud. What it actually takes — in RAM, bandwidth, power and BOM — to ship one in a device.

By Axon Labs Engineering

In 2026, small language models crossed from demo to deployable: Microsoft’s Phi-4-Mini quantized to 4-bit fits in roughly 1.2 GB, down from 7.6 GB in FP16, and runs interactively on phone-class silicon. The coverage stops at phones and laptops, though. The harder question — the one we get in feasibility studies — is whether a product can run one: a device with a BOM ceiling, a battery, and no fan. The answer is yes, more often than you’d think, but the gate isn’t the one the datasheets advertise.

Key takeaways

  • 2026-generation small language models (0.5–14B parameters) run real product features on device: Phi-4-Mini at 4-bit fits in ~1.2 GB, Gemma 4 E2B in ~1.5 GB.
  • Decode is memory-bound, not compute-bound: mobile-class silicon moves 50–90 GB/s against 2–3 TB/s in the data center, so tokens-per-second follows bandwidth — TOPS is the wrong spec to shop by.
  • Ship the cascade — always-on wake word, on-device SLM for commands and dialogue, cloud for the long tail — and validate latency, memory and accuracy on the target silicon as a phase gate.

What is a small language model — and why did 2026 change the answer?

A small language model is a 0.5–14 billion-parameter model built to run on constrained hardware rather than a server rack (CogitX, 2026 SLM guide). In 2026 the tier matured into a deep bench — Microsoft’s Phi-4 family, Google’s Gemma, Alibaba’s Qwen3, Meta’s Llama 3.2 — with aggressive quantization doing the heavy lifting: 4-bit weights cut Phi-4-Mini from 7.6 GB to about 1.2 GB with usable quality (local-LLM optimization guide, 2026).
For products, that changes the feasibility math. A language interface — spoken commands, on-device Q&A over the manual, structured dialogue — no longer requires a cloud round trip, a subscription COGS line, or shipping the user’s voice off the device. It requires gigabytes, not racks. The question becomes which gigabytes, and that’s where most datasheet reasoning goes wrong.

The real gate is memory, not TOPS

Generating text is a memory exercise: for every token produced, the runtime streams essentially the entire weight file through the processor. Decode is therefore bandwidth-bound — the compute units idle while weights load. Mobile-class devices move 50–90 GB/s; data-center GPUs move 2–3 TB/s, a 30–50× gap that no NPU rating closes (LLM Inference at the Edge, arXiv 2026).
The uncomfortable corollary, as of mid-2026: mainstream runtimes — llama.cpp, Ollama, LM Studio — still route language workloads to the CPU or GPU, not the NPU (MayhemCode, 2026). The TOPS number that sells the chip mostly accelerates vision and prefill, not the token-by-token decode your user experiences. For language on a device, first ask: how many bytes is the model, and how fast can this board move bytes?

What fits where: silicon classes for language

The three embedded silicon classes from our edge AI development guide map onto language workloads unevenly — and the bottom class doesn’t map at all.
Silicon classRealistic language capabilityWhat it enables
MCU-class accelerator (<1 W)Keyword spotting, intent grammars — tens of millions of parameters, not billionsWake words, fixed command sets. Not language generation.
NPU module / mid-range SoC (1–10 W, 2–4 GB RAM)0.5–4B models at 4-bitSpoken commands, structured dialogue, on-device Q&A over product docs
High-end edge SoC (10–30 W, 8 GB+)4–14B models at 4-bitAssistant-grade conversation, summarization, multimodal pipelines
Treat the middle row as the product sweet spot in 2026: a quantized 1–3B model on a Linux-class SoC with LPDDR4X. It’s the cheapest configuration where a language feature stops feeling like a demo. The ranges are indicative — validate against your workload, because tokenizer overhead, context length and thermal limits move the lines.

What does a language interface cost the BOM?

In BOM terms, an SLM is mostly a memory purchase. The weights need residence — 1–2 GB for the model itself at 4-bit — plus the KV cache that stores conversation context and grows linearly with context length, plus the OS and your application. Practical floor for the middle-row experience: a 4 GB-RAM device, with the model on eMMC/UFS storage.
  • RAM is the new BOM line. The delta from 1 GB to 4 GB of LPDDR4X is dollars — real, but small against the cloud-inference subscription it replaces over a device lifetime.
  • Power is per-interaction, not always-on. Decode is a burst workload; duty-cycle it behind a wake word and the battery story survives. Sustained conversation is where thermals bite — a fanless enclosure at multi-watt decode needs the thermal design done early.
  • The cloud line never fully disappears. Keep it for the long tail, and price the on-device tier as what it is: latency, privacy and COGS insurance.

Local, hybrid, or cloud: the placement test

The placement question for language is the same one we apply to every intelligent-device architecture: decisions needed in under a second, on battery, or offline belong on the device; capabilities that improve with unbounded context belong in the cloud. For language, that resolves into a three-tier cascade.
The wake-word tier runs always-on at microwatts on MCU-class silicon. The SLM tier handles the product’s actual vocabulary — commands, settings, questions about itself — locally, deterministically, offline. The cloud tier catches the long tail, when connectivity allows and the user consents. This is the same hybrid split that held our smart insole to a sub-$50 BOM: modest silicon doing the always-on work, heavier intelligence invoked only when it earns its power budget.

A talking product is a regulated product

Two EU regimes now assume your language feature exists. From 2 August 2026, the AI Act’s Article 50 requires products whose AI interacts directly with people to disclose it — at first interaction, visibly; a line in the terms doesn’t count (Article 50, Regulation (EU) 2024/1689). And from 11 September 2026, the Cyber Resilience Act’s reporting obligations apply to every connected product — with model files joining firmware as assets that need signed, revertible update paths.
The engineering consequence: build the disclosure into the onboarding flow, and treat the model as a versioned, updatable artifact inside the same secure-OTA architecture the CRA already demands. One architecture, both regimes — designed in, not bolted on.

The model and compression playbook

  • Pick for the task, not the leaderboard. A 1B model fine-tuned on your product’s domain beats a general 7B model that doesn’t fit your RAM. The 2026 bench — Phi-4, Gemma, Qwen3, Llama 3.2 — all ship small variants with permissive licensing tiers (BentoML, 2026); check the license against your commercial use before falling in love.
  • Quantize to 4-bit as the default. It’s the difference between fitting and not fitting; recover quality with quantization-aware fine-tuning on your domain data — the same prune-quantize-distill stack from our edge AI guide applies.
  • Budget the context window like power. KV cache grows linearly with context; an unbounded chat history is a memory leak with a UX. Decide the context your feature actually needs and cap it.
  • Choose a runtime you can ship. llama.cpp for portability, ExecuTorch and vendor SDKs where NPU support is real for your chip. Verify claimed acceleration on your board — mid-2026, much of it isn’t wired up yet.
  • Validate on target, as a phase gate. Tokens/second, memory high-water mark, thermal steady-state and task accuracy on the actual board, with pass/fail thresholds at EVT — the same evidence discipline as every other subsystem in hardware product development.
Budget the memory bandwidth, not the TOPS.

The bottom line

  • Yes, your product can probably run a language model in 2026 — a quantized 0.5–4B SLM on a 4 GB-RAM SoC is the practical floor, and it removes cloud latency, connectivity dependence and per-query COGS.
  • Shop by bytes and bandwidth, not TOPS: decode speed ≈ memory bandwidth ÷ model size, and mainstream runtimes still bypass most NPUs for language.
  • Ship the cascade — wake word, local SLM, cloud long tail — disclose the AI at first interaction, and put model updates inside the signed-OTA architecture the CRA already requires.
If a language interface is on your product roadmap, the feasibility math above takes us about a week to run against your BOM, power budget and silicon shortlist. Start with a discovery & feasibility phase — cheaper than finding out at EVT that the memory bus was the spec that mattered.

Frequently asked questions

Can a microcontroller run an LLM?

Not meaningfully. MCU-class accelerators handle keyword spotting and intent models in the tens of millions of parameters — language generation starts at NPU-module or SoC class with gigabytes of RAM. The right MCU role is the always-on wake-word tier that gates a bigger model.

How much RAM does an on-device language model need?

Model weights plus KV cache plus system. A 2026-generation SLM at 4-bit occupies 1–2 GB (Phi-4-Mini ~1.2 GB; Gemma 4 E2B ~1.5 GB), and the KV cache grows linearly with context length — making a 4 GB-RAM device the practical floor for a product-grade language feature.

How fast is on-device language inference in 2026?

Bandwidth-bound: roughly 1–5 tokens/second on consumer or edge-class CPUs and 10–20 tokens/second for an 8B model on a good desktop CPU, versus 10–100 on GPUs. Conversational speech needs only ~3–5 tokens/second, which well-matched SLM-and-silicon pairings achieve.

Which small language models fit embedded products in 2026?

The main families all ship device-class variants: Microsoft Phi-4 (including Phi-4-Mini), Google Gemma, Alibaba Qwen3 and Meta Llama 3.2, typically 0.5–4B parameters quantized to 4-bit. Choose by task fit, RAM footprint, license terms and runtime support for your silicon — in that order.

Do NPU TOPS make language models faster?

Mostly not, as of mid-2026. Token generation is memory-bandwidth-bound, and mainstream runtimes (llama.cpp, Ollama, LM Studio) still route language workloads to CPU or GPU rather than the NPU. TOPS ratings chiefly accelerate vision and prompt prefill — for decode speed, divide memory bandwidth by model size.

Sources

If the product has to ship, talk to the team that builds for that outcome.

Senior engineer on the first call. NDA before technical detail. References available under NDA after qualification. Or start with a fixed-fee feasibility study.

Request a conversationFeasibility study