Retrospectively Reverse-Engineering Apple's Neural Engine
236 points
• 2 days ago
• Article
Link
Apple Neural Engine（ANE）最初被设计为针对密集卷积神经网络（CNN）的专用加速器，这类网络依赖于可预测的数据重用。通过大规模并行的乘加（MAC）单元阵列和优化的数据流路径，硬件能高效处理这些模型。但随着行业工作负载转向 Transformer，ANE 对固定数据移动和专用本地内存结构的依赖成为瓶颈——解码的自回归特性打破了其设计所依赖的可预测重用假设。
在计算核心层面，ANE 采用 16 个核心，每个核心配备 128 条 FP16 或 256 条 INT8 的 MAC 通道，总计 2048 条并行通道。每条通道随时间执行标量归约，并通过反馈路径将部分和保存在本地内存中。对于 CNN 用例，这种设计非常高效，因为它减少了与主内存的频繁往返。非线性激活（如 tanh 或 ReLU）由集成的后 MAC 处理模块负责，该模块使用 33 项的分段线性查找表。通过在编译阶段将常量缩放和偏置合并到卷积中，硬件把这些层的开销降到了最低。
调度机制高度静态，更像固定功能引擎而非可编程处理器。驱动软件不会下发单独的神经算子，而是将模型编译成一串任务描述符（TD），通过 doorbell 提交。任务描述符相当于序列化的寄存器文件转储，用于配置某次运算的数据通路。由于没有通用指令集，ANE 只能采用 Apple 暴露的数据通路配置，因此实际上需要定制编译器来精细编排张量的移动与变换。
从 roofline 模型看，内存吞吐量仍然是 ANE 的主要瓶颈。尽管 M1 上的 ANE 峰值算力颇具竞争力，但它受限于内存层次结构，数据必须从主 DRAM 流入本地 L2 和内核内存（KMem）银行。用于内核权重和输入切片的 DMA 引擎似乎以串行方式工作，无法饱和系统的总内存带宽，因此在像 Transformer token 解码这类高度依赖大量、快速数据移动的负载上难以与 GPU 抗衡。
总的来说，ANE 的专用化设计反映了对特定机器学习时代的硅片效率追求。随着诸如 M5 的新芯片逐渐将 ANE 功能直接并入 GPU 核心，作为独立且强烈有偏好的 NPU 的时代正逐步衰退。虽然通过巧妙的编译时调度该硬件在技术上仍能运行现代 Transformer，但其架构僵化和 DRAM 吞吐限制表明，未来的性能提升更可能来自更广泛、更统一的计算平台，而非这些高度优化却僵化的 CNN 时代孤岛。
The Apple Neural Engine (ANE) was originally architected as a specialized accelerator for dense convolutional neural networks (CNNs), which rely on predictable data reuse patterns. By utilizing large parallel arrays of multiply-accumulate (MAC) units and optimized dataflow paths, the hardware could efficiently process these models. However, the architecture's reliance on fixed data movement and specialized local memory structures eventually created a bottleneck as industry workloads shifted toward transformer models, where the autoregressive nature of decoding breaks the assumptions of predictable reuse that the ANE was designed to exploit.
At the compute core level, the ANE utilizes 16 cores, each equipped with 128 FP16 or 256 INT8 MAC lanes, totaling 2048 parallel lanes. Each lane performs a scalar reduction over time, with a feedback path that keeps partial sums in local memory. This setup is highly efficient for its intended CNN use case, as it avoids constant round-trips to main memory. Nonlinear activations, such as tanh or ReLU, are handled by an integrated post-MAC block that uses a 33-entry piecewise-linear lookup table. By folding constant scaling and bias operations into the convolution phase during compilation, the hardware minimizes the overhead associated with these layers.
The scheduling mechanism is surprisingly static, functioning less like a programmable processor and more like a fixed-function engine. The driver software does not issue individual neural operations; instead, it compiles models into a stream of task descriptors (TDs) that are submitted via a doorbell mechanism. A task descriptor serves as a serialized register-file dump that configures the datapath for a specific pass. Because there is no general-purpose instruction set architecture, the ANE is strictly limited to the datapath configurations exposed by Apple, effectively requiring a custom compiler to orchestrate every detail of how tensors move and transform.
Memory throughput remains the primary constraint for the ANE, particularly given the roofline model of its operation. While the M1 ANE boasts significant peak performance, it is bound by a memory hierarchy where data must be streamed from main DRAM into local L2 and kernel memory (KMem) banks. Because the DMA engines for kernel weights and input tiles appear to operate serially and cannot saturate the system's total memory bandwidth, the hardware struggles to compete with GPUs for workloads like transformer token decoding, which are heavily dependent on massive, rapid data movement.
Ultimately, the ANE's specialized design represents a commitment to silicon efficiency for a specific era of machine learning. The fact that newer chips, such as the M5, have begun folding ANE functionality directly into the GPU cores suggests that the era of the standalone NPU as an independent, opinionated accelerator is waning. While the hardware remains technically capable of running modern transformers through clever compile-time scheduling, its architectural inflexibility and limitations in DRAM throughput suggest that future performance gains will rely on broader, more unified compute platforms rather than these highly optimized, yet rigid, CNN-era silos.
33 comments • Comments Link
• M4 Neural Engine (ANE) 引入了对快速路径 INT8 权重和激活的专门支持，后续的芯片（如 M5 Ultra 和 M6）则通过采用双 ANE 模块来扩展性能。
• ANE 的架构不同于新型 GPU 中的 Neural Accelerator (NAX) 核心，后者更接近类似 NVIDIA tensor cores 的传统矩阵乘法单元。
• Apple 早在 2017 年就将 Neural Engine 集成进产品，证明了其在硬件加速机器学习方面的长期战略投入，这早于当前生成式 AI 流行的浪潮。
• 硬件未来的验证本身就很困难，因为芯片设计周期明显长于软件演进，这使得提前数年预测像 Transformers 这样的主流架构变得极具挑战性。
• ANE 的最初设计对 Convolutional Neural Networks (CNNs) 进行了深度优化，这很可能源于早期对自动驾驶技术的研究，当时计算机视觉是主要需求。
• 与 ANE 配合的开发者不得不创造性地将现代基于 Transformer 的工作负载映射到其以 CNN 为中心的硬件流水线上，通常的做法是将 4D tensors 当作图像数据，并用 1x1 convolutions 替代 matmuls 。
• 由 Apple 控制的生态系统在本地运行高性能 AI 模型时代具有天然优势，因为它允许用户在本地硬件上运行私有且参数量大的模型，而不必完全依赖昂贵的云端订阅。
• 尽管有人认为 ANE 没有被充分利用，但多年来它一直在为关键后台任务提供算力支持，包括 Face ID 、 crash detection 、 voice isolation 以及 Photos app 中的本地图像分析。
• 即将推出的 Core AI 框架旨在弥补旧版 Core ML 的不足，使应用能更好地协调使用 CPU 、 GPU 和 Neural Engine 来运行当代模型架构。
• AI 快速发展的步伐对传统科技巨头构成潜在颠覆：随着硬件和软件门槛降低，创新可能在既有参与者之外更广泛地爆发。
这些讨论反映出硬件加速机器学习的长期愿景与 AI 软件架构快速且不可预测演进之间的张力。 ANE 最初为定义过去十年计算机视觉与 CNN 任务而优化，但其向支持现代 Transformer 工作负载的转变也凸显出要把固定硬件与不断变化的软件范式匹配起来有多难。尽管有人批评其影响力不够显眼，ANE 多年来却默默支撑着 Apple 生态中的核心功能。展望未来，行业可能会被本地、高效的推理所重塑，在这种模式下，能够弥合旧有设计与现代需求差距的专用硬件将成为重要的竞争差异点。 • The M4 Neural Engine (ANE) introduced specialized support for fast-path INT8 weights and activations, while subsequent chips like the M5 Ultra and M6 feature dual ANE blocks to scale performance.
• The ANE architecture is distinct from the Neural Accelerator (NAX) cores found in newer GPUs, which function more like traditional matrix multiplication units similar to NVIDIA tensor cores.
• Apple's early integration of the Neural Engine in 2017 demonstrates a long-term strategic investment in hardware-accelerated machine learning that predates the current wave of generative AI popularity.
• Future-proofing hardware is inherently difficult because silicon design cycles are significantly longer than software evolution, making it challenging to anticipate dominant architectures like Transformers years in advance.
• The original design of the ANE was heavily optimized for Convolutional Neural Networks (CNNs), likely stemming from early research into self-driving technology where computer vision was the primary requirement.
• Developers working with the ANE have had to creatively map modern Transformer-based workloads onto its CNN-centric hardware pipeline, often by treating 4D tensors as image data and replacing matmuls with 1x1 convolutions.
• The Apple-controlled ecosystem is potentially well-positioned for an era of local, high-performance AI models, as it allows users to run private, parameter-heavy workloads on local hardware rather than relying solely on expensive cloud-based subscriptions.
• While some argue the ANE has been underutilized, it has powered critical background tasks for years, including Face ID, crash detection, voice isolation, and local image analysis within the Photos app.
• The upcoming "Core AI" framework aims to bridge the gap left by the older Core ML, enabling applications to better utilize the CPU, GPU, and Neural Engine for contemporary model architectures.
• The rapid pace of AI advancement poses a potential disruption to established tech giants, as lowered barriers for hardware and software development may trigger a broader explosion of innovation beyond traditional players.
The discussion reflects a tension between the long-term vision of hardware-accelerated machine learning and the rapid, unpredictable evolution of AI software architectures. While the Apple Neural Engine was initially optimized for the computer vision and CNN tasks that defined the previous decade, its pivot to support modern Transformer-based workloads illustrates the inherent difficulty of aligning fixed hardware with shifting software paradigms. Despite criticisms regarding the visibility of its impact, the hardware has quietly underpinned core functional features in Apple's ecosystem for years. Looking forward, the discourse suggests that the future of the industry may be reshaped by local, efficient inference, where specialized hardware that bridges the gap between older designs and modern requirements becomes a significant competitive differentiator.