Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
277 points
• 5 days ago
• Article
Link
Deltafin 是一个实验性开源项目，目标是在普通消费者硬件上本地运行完整的 2.8-trillion-parameter Kimi K3 模型。不同于通过剪枝或量化权重换取性能的做法，Deltafin 更重视保持 MoE (Mixture of Experts) 模型的原始质量。通过使用单一的 native binary，Deltafin 确保每个生成的 token 都由 Kimi K3 作为唯一权威决定，较小的 draft models 仅作为猜测器，其输出必须由主模型显式验证。
目前的实现侧重极致的硬件效率，特别针对配备 128 GB 内存并用多块 SSD 流式加载 expert weights 的 Apple Silicon 平台。基准测试显示，在 M5 Max MacBook Pro 上，Deltafin 在稳定解码时约能达到 1.00 token per second 。性能高度依赖存储吞吐，数据表明增加驱动器收益递减，因为每层需要读取的 16 个 expert reads 中最慢的那一项决定了总体速度。
虽然项目在优化上已有显著进展，但在一些技术难题上仍需突破，尤其是初始响应速度。目前由于在 prefill 阶段对模型层进行了重复读取，一个 512-token 的 prompt 在生成首个 token 前会有超过六分钟的延迟。开发者认为这是存储与架构上的瓶颈，而非硬件本身的限制，计划在后续迭代中通过优化来缩短这一延迟。
项目设计上提供了灵活的部署方式，既支持 full 安装，也支持 stream 模式。 stream 模式让用户以更小的存储起步，并在使用过程中逐步在本地缓存 experts 。 Deltafin 还内置了一个与 OpenAI 兼容的服务器，便于与各种客户端集成，并强制执行严格、贪婪且可重现的生成流程，以保证与模型预期输出一致。
总之，Deltafin 致力于在家用基础设施上拓展可能性。通过示范如何在不牺牲模型完整性的前提下运行大规模模型，项目希望为更广泛的 local-AI 社区做出贡献。开发者强调其核心任务是探索，并记录每次实验与基准测试，以透明披露在可及的高端消费者系统上运行 frontier models 的成本与能力。
Deltafin is an experimental, open-source project designed to run the full, 2.8-trillion-parameter Kimi K3 model locally on consumer hardware. Unlike other approaches that prune or quantize model weights to achieve performance, Deltafin prioritizes maintaining the original quality of the MoE (Mixture of Experts) model. By leveraging a single native binary, it ensures that Kimi K3 remains the sole authority for every generated token, with small draft models acting only as guessers that require explicit verification by the main model.
The current implementation focuses on extreme hardware efficiency, specifically targeting Apple Silicon setups with 128 GB of RAM and multiple SSDs for streaming expert weights. Benchmarks indicate that, when using an M5 Max MacBook Pro, Deltafin achieves approximately 1.00 token per second for steady decoding. The performance is heavily reliant on storage throughput, with data showing that adding more drives yields diminishing returns, as the total speed is constrained by the slowest of the 16 expert reads required for each layer.
While the project has made significant strides in optimization, there are still technical hurdles to overcome, particularly regarding initial processing speeds. A 512-token prompt currently faces a delay of over six minutes before the first token is generated due to redundant re-reads of model layers during the prefill phase. The developers have identified this as a storage and architectural bottleneck rather than a fundamental limitation of the hardware, and they have planned optimizations to address this latency in future iterations.
The project is structured to offer flexibility in how users deploy the model, providing both a "full" installation and a "stream" mode. The streaming option allows users to start with a smaller storage footprint and build a local cache of experts over time as they interact with the model. Additionally, Deltafin includes an OpenAI-compatible server, which allows it to integrate with various clients while enforcing a strict, greedy, and reproducible generation process that mirrors the model's intended output.
Ultimately, Deltafin serves as a research effort to push the boundaries of what is possible on home infrastructure. By demonstrating that large-scale models can be executed without compromising their integrity, the project aims to contribute to the broader local-AI community. The developers emphasize that their primary mission is exploration, documenting every experiment and benchmark to provide transparency into the costs and capabilities of running frontier models on accessible, albeit high-end, consumer systems.
155 comments • Comments Link
• 在 MacBook Pro 配合外接 Thunderbolt SSD 等消费级硬件上，实现了对 2.78T 参数模型 Kimi K3 的每秒 token 推理，证明了在本地运行前沿规模模型的可行性。
• 当前系统架构需要从磁盘流式读取专家权重，这在预填充阶段形成严重瓶颈——每层需读取约 9TB 数据，极大地占用内存带宽。
• 性能扩展高度依赖磁盘吞吐量。所谓"drive ladder"显示，随着系统中 SSD 数量增加，收益递减，说明顺序带宽与调度策略比单纯增加硬件更为关键。
• 虽然当前推理速度对交互式聊天而言过慢，但这种配置适用于异步、无人值守的任务——模型可在夜间批量处理数据，从而实现数据不出机的本地高参数模型运行。
• 本地 LLM 性能的未来可能依赖定制的 ASIC 架构，例如晶圆级芯片，将极高的内存带宽置于目前 GPU 常见的商品化离散 DRAM 之上。
• 关于模块化 LLM 的设想颇多：通过索引"粘性"专家或权重，让模型的大量参数只需有一小部分驻留在 VRAM 中，从而有效降低万亿参数模型的使用门槛。
• 社区内部存在明显分歧：一方面优先考虑即时的实际应用，另一方面则更看重技术探索本身以及"试试看能不能做到"的精神。
• 在预填充调度方面做出重大优化（将每层多次读取专家改为单次顺序通过）是让这些"缓慢"的本地模型可用于诸如单 token 分类等任务的主要途径。
• 随着 AI 模型不断增强，编码与测试的工作流正在向人机协作转变，打字和调试的"成本"正在被迭代提示的成本所取代。
• 尽管对在本地运行模型存在普遍怀疑，但出于隐私保护、避免经常性订阅开支以及在本地硬件上利用大规模模型的需求，相关技术实验仍然受到强烈驱动。
讨论的核心在于：在消费级硬件上尝试运行万亿级参数模型的技术胆识，与当前速度限制之间的矛盾。参与者围绕这种缓慢推理的实际效用展开辩论，最终在强调本地数据主权而非实时响应的无人值守批处理工作流中找到了价值。这项努力既被视为对即时生产力的探索，也被看作迈向模块化、本地化 AI 的重要一步，呼应了计算史上"过去认为不可能的事情最终成为常态"的发展轨迹。 • Achieving token-per-second inference on a 2.78T parameter model (Kimi K3) using consumer hardware like a MacBook Pro and external Thunderbolt SSDs demonstrates the feasibility of running frontier-scale models locally.
• The current system architecture involves streaming expert weights from disk, which creates significant bottlenecks during the prefill phase where memory bandwidth is consumed reading 9TB of data per layer.
• Performance scaling is heavily dependent on disk throughput, with a "drive ladder" showing diminishing returns as more SSDs are added to the system, proving that sequential bandwidth and scheduling are more critical than raw hardware count.
• While the current inference speed is too slow for interactive chat, the setup is viable for asynchronous, unattended tasks where a model processes data overnight, allowing for local execution of high-parameter models without data leaving the machine.
• The future of local LLM performance likely involves custom ASIC architectures, such as wafer-scale chips, which prioritize extremely high memory bandwidth over the commodity discrete DRAM found in current GPUs.
• Speculation exists regarding modular LLM design, where "sticky" experts or weights could be indexed to allow only a fraction of a massive model to reside in VRAM, effectively democratizing the use of trillion-parameter models.
• There is a clear divide in the community between those who prioritize immediate practical application and those who value the spirit of technical exploration and "just seeing if it can be done."
• Significant optimizations in prefill scheduling—moving from reading experts multiple times per layer to a single-pass approach—represent the primary path toward making these "slow" local models useful for tasks like single-token classification.
• Coding and testing workflows are shifting as AI models become more capable, with a trajectory leading toward human-AI collaboration where the "cost" of typing and debugging is replaced by the cost of iterative prompting.
• Despite common skepticism about running models locally, the ability to maintain privacy, avoid recurring subscription costs, and utilize massive models on local hardware remains a compelling motivation for technical experimentation.
The discussion centers on the tension between the technical audacity of running trillion-parameter models on consumer hardware and the practical limitations of current speed benchmarks. Participants debate the utility of such slow inference, ultimately finding value in unattended, batch-style workflows that emphasize local data sovereignty over real-time responsiveness. This effort is framed not merely as a quest for immediate productivity, but as a fundamental step toward modular, local AI, mirroring the historical trajectory of computing where previously "impossible" tasks eventually became standard.