What is an NPU? How does it differ from CPU and GPU?
What is an NPU?
A Neural Processing Unit (NPU) is a class of specialized hardware accelerator built for artificial intelligence (AI) and machine learning (ML) workloads. It is also known as an AI accelerator or deep learning processor. An NPU targets neural network inference, training, and machine vision tasks. It can be a standalone chip, a dedicated block inside a system-on-chip (SoC), or an integrated logic unit within a CPU or GPU.
NPU architectures are optimized for the mathematics of deep learning: tensor operations, large matrix multiplications, convolutions, and element-wise activations. They achieve much higher throughput and far lower power consumption than general-purpose processors on these tasks. Design features include manycore or spatial arrays, low-precision arithmetic (INT4, INT8, FP8, FP16), dataflow execution models, and in-memory computing that minimises data movement between memory and compute.
On the consumer side, Apple’s Neural Engine in the M-series and A-series chips, Qualcomm’s Hexagon NPU in Snapdragon, Samsung’s in-house NPU, Huawei’s Da Vinci architecture, and Google’s Tensor Processing Unit in Pixel devices all fit this description. Intel’s Meteor Lake processors (2023) introduced a dedicated NPU branded Intel AI Boost, originally conceived as a Vision Processing Unit. Data-centre examples include Google’s cloud TPU pods, AWS Trainium and Inferentia, and the specialised matrix accelerators inside NVIDIA and AMD GPUs. Almost all modern GPU silicon dedicates die area to low-precision matrix engines, but the NPU label is reserved for hardware that is built from the ground up around neural network computation.
How does an NPU differ from a CPU and GPU?
A central processing unit (CPU) is a generalist. It has a handful of high-performance cores, sophisticated control logic, deep cache hierarchies, and wide register files designed for serial, latency-sensitive tasks. CPUs support high-precision arithmetic (FP64, FP32) and fast context switching. They handle everything from operating system code to spreadsheets but are inefficient when performing the same simple math on thousands of data points simultaneously.
A graphics processing unit (GPU) is a parallel worker. It contains hundreds or thousands of smaller compute units organised for massive throughput under a Single Instruction Multiple Data (SIMD) model. GPUs excel at graphics shading, scientific simulations, and the matrix multiplications used in deep learning. Modern GPUs include tensor cores (NVIDIA) or matrix cores (AMD) that accelerate low-precision multiply-accumulate operations, blurring the line with NPUs. However, a GPU must still support graphics pipelines and general-purpose parallel kernels, so it consumes significantly more power for a pure AI workload compared with a dedicated NPU.
An NPU is a specialist. Its execution units, data paths, and memory are designed exclusively for neural network dataflows. Common design patterns include systolic arrays and spatial compute fabrics that multiply and accumulate thousands of values each cycle without moving intermediate results to external memory. This specialisation yields vastly better performance per watt for AI inference. An analogy: a CPU is a small team of expert surgeons performing delicate operations one after another; a GPU is a giant army of clerks each copying numbers simultaneously; an NPU is a purpose-built assembly line producing exactly one product at blistering speed.
A Vision Processing Unit (VPU) is a subset of NPU, aimed specifically at machine vision algorithms such as convolutional neural networks and scale-invariant feature transform (SIFT). VPUs are found in augmented reality headsets, smart cameras, and drones. Programming NPUs relies on high-level frameworks: TensorFlow Lite/RT (Android), CoreML (Apple), DirectML (Windows), and the ONNX interchange format. Vendor-specific APIs like Intel OpenVINO, Qualcomm SNPE, AMD Ryzen AI, and Apple CoreML expose low-level control. GPUs use CUDA, OpenCL, or Vulkan for AI, which gives more flexibility but demands more developer effort and often higher energy.
What is TOPS and how is NPU performance measured?
TOPS stands for Trillions of Operations Per Second. It is the simplest headline metric for AI accelerator performance. A single “operation” is almost always a multiply-accumulate (MAC), the fused multiply-add at the heart of matrix multiplication. The number is reported at a specific precision, usually INT8 or FP16, because wildly different figures can be quoted for lower-bit precisions. For instance, Apple’s M4 Neural Engine is rated at 38 TOPS (INT8), Qualcomm’s Snapdragon X Elite reaches 45 TOPS, Intel Meteor Lake’s NPU delivers up to 11.5 TOPS, while an NVIDIA H100 GPU can hit over 3,900 TOPS (INT8, with sparsity).
TOPS is a peak theoretical figure. Sustained performance depends on model architecture, memory bandwidth, software runtime efficiency, and support for structural sparsity. Two accelerators with the same TOPS rating can show markedly different inference speed in real applications. Benchmark suites such as MLPerf provide standardized measurements that account for these factors. When comparing NPUs, look at TOPS for the precision your model will actually use—many on-device inference workloads use INT8 to reduce memory and power, so INT8 TOPS is the most relevant consumer number.
What are NPUs used for in everyday devices?
NPUs accelerate specific AI workloads that would otherwise run slowly or drain the battery on a CPU or GPU. In smartphones, Apple introduced its Neural Engine with the A11 Bionic chip in 2017. This dedicated block handles Face ID recognition, animoji tracking, and computational photography tasks such as Smart HDR and Deep Fusion. Scene detection in the camera app now identifies objects, pets, food, and landscapes in real time to adjust exposure and color before the shutter fires. Voice recognition for Siri runs entirely on the device when an internet connection is unavailable, and the same local processing powers real-time translation in the Translate app. Augmented reality applications use the Neural Engine to track planes and objects without jitter, keeping virtual overlays stable as the user moves the phone.
Qualcomm integrates a Hexagon NPU into its Snapdragon platforms. The tensor accelerator inside the Hexagon block runs always-on voice assistants, echo cancellation, and noise suppression during calls. Samsung’s Exynos chips include a dedicated NPU for similar workloads: camera scene optimization, secure face or iris recognition, and Bixby voice commands offline. Google’s Tensor chip has a custom TPU core (Tensor Processing Unit) originally derived from its data center designs. That core powers on-device Live Caption, Recorder app transcription without the cloud, and Magic Eraser in Google Photos. All these examples share a common thread: the NPU processes deep learning models that were once too heavy for mobile processors, and it does so while consuming a fraction of the energy a GPU would need for the same task.
Laptops are using NPUs for video conferencing and creative workflows. Intel’s Meteor Lake processors contain an NPU brand-named AI Boost. On a Windows laptop, this unit handles Windows Studio Effects: automatic background blur, eye contact correction that adjusts gaze so the user appears to look at the camera, and acoustic echo cancellation that isolates the speaker’s voice even in a noisy room. AMD Ryzen AI, embedded in Ryzen 7040 series and later, performs similar functions. Video editing software taps the NPU for scene detection—sorting footage into b-roll, interviews, or action clips—without maxing out the CPU or GPU. Adobe Premiere Pro and DaVinci Resolve now offload certain AI filters like object removal and style transfer to the NPU, keeping the timeline responsive. These laptops also support ONNX, an open format that lets developers exchange models between different frameworks and hardware accelerators, making it easier for an NPU from Intel, AMD, or Qualcomm to run the same trained neural network.
Why are smartphones and laptops now including NPUs?
The primary reasons are latency, privacy, and battery life. Running an AI model on a remote server adds round-trip time that degrades the user experience for real-time features. Face unlock would feel sluggish if every recognition attempt required a server request. Local processing on an NPU eliminates that delay. Privacy compliance is another driver. Voice assistants that process audio entirely on the device do not send recordings to the cloud, meeting stricter data protection rules and user expectations. Battery life improves because a dedicated NPU can execute a neural network at 10 to 20 times the energy efficiency of a CPU or GPU for the same performance level. A GPU is a parallel processor with hundreds of cores designed for graphics, and while it can run matrix multiplication—the backbone of neural network inference—it cannot match the energy efficiency of a fixed-function array of multiply-accumulate units tailored for low-precision integer math. The NPU’s architecture uses systolic arrays and sparsity exploitation to burn microwatts per inference, extending the time between charges.
A measurable benchmark is TOPS (Tera Operations Per Second), which measures integer inference throughput. Microsoft’s Copilot+ PC specification requires an NPU with at least 40 TOPS. The first wave of laptops meeting this requirement—using Qualcomm Snapdragon X Elite, AMD Ryzen AI 300 series, and Intel Lunar Lake—demonstrates that the industry now treats an NPU as essential, not optional. The NPU becomes a third pillar alongside the CPU and GPU in a heterogeneous system architecture (HSA). The Khronos Group, the consortium behind OpenGL and Vulkan, is standardizing AI APIs with the aim of releasing a unified NNEF (Neural Network Exchange Format) consumer standard in 2025. Standardization will let any application request an inference and let the driver schedule it on the most appropriate processor—NPU, GPU, or CPU—without developer intervention. This abstraction layer matters for laptops and phones because it removes the burden of hand-optimizing models for each vendor’s silicon, accelerating the rollout of new on-device AI features across the ecosystem.
Reference: Wikipedia
