Bonsai 27B: A 27B-Class model that runs on a phone
696 points
• 2 days ago
• Article
Link
PrismML 宣布发布 Bonsai 27B，这是一款基于 Qwen3.6 的强大多模态模型，标志着端侧 AI 的一个重要里程碑。此前已有模型表明低位权重可以有效，但 Bonsai 27B 是同类能力中首个能在手机上运行的版本。这一成就将多步推理、结构化工具调用和复杂的代理循环等高级功能带到本地硬件上——这些功能以前因 27B 参数模型所需的大量内存而难以在设备上实现。
本次发布包含两个针对不同性能与资源需求的版本。 Ternary Bonsai 27B 采用三值权重，使得每个权重的有效位数约为 1.71 比特，占用约 5.9 GB 内存，旨在为普通笔记本提供高质量的推理与代理能力。 1-bit Bonsai 27B 采用二值权重，每个权重有效位数约为 1.125 比特，将模型体积缩至约 3.9 GB，足够在 iPhone 17 Pro 的内存限制内运行，体现了在单位内存中提升智能密度的重大突破。两款模型均支持 262K-token 的上下文窗口，并通过推测性解码提高运行速度。
在性能上，Bonsai 27B 系列与全精度模型非常接近。在覆盖 15 项基准的测试中，Ternary 模型保持了基线能力的 95%，1-bit 版本保持了约 90% 。尤其是数学、编程和工具调用等核心能力仍然很强。这种能力保留使得模型可以处理需要持续且精准推理的复杂代理工作流，有效缩小云端任务与本地处理之间的差距。
向本地执行的转变解决了云端 AI 的若干限制，尤其对在单次工作流中执行数百步操作的代理来说意义重大。设备原生运行可以消除网络请求带来的延迟与费用，同时确保敏感用户数据（如私人文件和屏幕内容）保留在本地硬件。该架构还支持混合部署：私密或高频任务在本地处理，而更大算力需求的前沿任务则可上云执行。
Bonsai 系列的方法论把"智能密度"作为未来 AI 发展的关键指标，通过在有限内存中尽量装入更多能力，PrismML 致力于让高端 AI 在更广泛的场景中可用。这些模型以 Apache 2.0 License 提供，并针对通过 MLX 运行的 Apple silicon 以及使用定制低位 CUDA 内核的 NVIDIA GPU 进行了优化。随着公司持续改进压缩技术，预计会不断推动日常设备上 AI 能力的边界。
PrismML has announced the release of Bonsai 27B, a powerful new multimodal model based on Qwen3.6 that marks a significant milestone in on-device AI. While previous models have demonstrated that low-bit weights could be effective, Bonsai 27B is the first of its capability class to operate on a mobile phone. This achievement brings advanced features such as multi-step reasoning, structured tool calls, and sophisticated agentic loops to local hardware, which were previously impractical due to the massive memory requirements typically associated with 27B-parameter models.
The release includes two distinct variants designed for specific performance and footprint needs. The Ternary Bonsai 27B, which uses three-value weights to achieve an effective 1.71 bits per weight, occupies 5.9 GB. It is built to deliver high-quality reasoning and agentic capabilities on standard laptops. The 1-bit Bonsai 27B, using binary weights for an effective 1.125 bits per weight, shrinks the footprint to just 3.9 GB. This specific version is small enough to fit within the memory constraints of an iPhone 17 Pro, representing a notable breakthrough in intelligence density. Both models support a 262K-token context window and leverage speculative decoding for enhanced speed.
In terms of performance, the Bonsai 27B variants remain remarkably close to their full-precision counterparts. Across a broad 15-benchmark suite, the Ternary model retains 95% of the baseline intelligence, while the 1-bit version keeps 90%. Notably, core capabilities like mathematics, coding, and tool calling remain highly effective. This level of retention allows the models to handle complex agentic workflows where sustained, accurate reasoning is essential, effectively bridging the gap between cloud-dependent tasks and local processing.
The shift toward local execution addresses key limitations of cloud-based AI, particularly for agents that perform hundreds of steps in a single workflow. By running natively on the device, the system removes the latency and costs associated with network requests while ensuring that sensitive user data, such as private files and screen content, remains on the hardware. This architecture also supports hybrid deployments, where private or high-frequency tasks are handled locally while more demanding, frontier-level tasks are sent to the cloud.
The underlying methodology of the Bonsai series emphasizes intelligence density as a critical metric for future AI development. By focusing on how much capability can be packed into a limited memory footprint, PrismML is working to make high-end AI accessible across a wide range of environments. The models are available under the Apache 2.0 License and are optimized for both Apple silicon via MLX and NVIDIA GPUs using custom low-bit CUDA kernels. As the company continues to refine its compression techniques, it expects to keep pushing the boundaries of what is possible on everyday devices.
250 comments • Comments Link
• 用户对在消费级硬件上运行强大 27B 参数模型表现出浓厚兴趣，尤其是采用 1.58-bit 或 1-bit 的极端量化（例如三值量化），因为这些方法能显著降低内存需求。
• 关于极端量化的有效性存在争议；有人指出过度压缩可能导致智能下降、工具调用失败，甚至使模型陷入重复输出的"doom loops"。
• 三值量化（权重为 -1 、 0 、 1）正被区别于纯 1-bit 二值方法，因为前者可以通过在权重组中使用特定的缩放因子来获得更高的精度。
• 这类模型的实现需要专门的软件支持，像 LM Studio 或 vanilla llama.cpp 这样的常规模型工具可能尚未完全支持这些高度压缩格式所需的特殊架构或自定义内核。
• 一些观察者对新的量化基准持怀疑态度，指出报告的结果往往依赖特定评测套件，可能无法反映相较于原始、低压缩模型的真实表现或整体"氛围"一致性。
• 外界对小型 AI lab 的商业策略颇感好奇，有猜测认为 Samsung 等大厂可能在资助这些团队，旨在将能在设备端运行的 AI 部署出来以对抗 Apple 的集成优势。
• 人们正在区分通用 LLMs 与专用的小规模模型；有观点认为，对于某些移动端任务，经过微调的较小模型优于被严重压缩的大型通用模型。
• 社区正积极构建个人化的评估流程来验证性能声明，这反映出对营销材料的不信任，以及对在不同量化方案下统一评测指标的需求。
• 关于开发者与 Apple 等大公司的潜在会谈被提及后，引发了企业机密性与当前 AI 研究开源性之间的紧张讨论。
• 有人对沟通质量表示担忧，认为最近的 AI 公告中的技术写作过于营销化（"marketing-speak"）或疑似由 AI 生成，因而分散了对潜在技术成果的注意力。
向极端量化的转变代表了一项重大技术攻关，目的是把大规模智能嵌入到移动设备或消费级笔记本等受限环境中。尽管在本地运行高参数模型的潜力令人期待，但在软件兼容性、模型可靠性以及参数规模与推理能力间的权衡方面仍存在显著障碍。总体共识认为，这些方法作为边缘计算的一个有前景方向值得关注，但当前在稳定性方面仍显脆弱，不太可能立刻成为日常使用的主力助手。社区正在转向更为批判且以数据为驱动的评估方法，倾向于依靠严谨的内部基准测试，而非轻信新研究小组最初常显得过于乐观的声明。 • Users are highly interested in the potential for running powerful 27B-parameter models on consumer hardware, particularly with 1.58-bit or 1-bit ternary quantization methods that significantly reduce memory requirements.
• There is a debate regarding the efficacy of extreme quantization, with some noting that compressing models too aggressively can lead to degraded intelligence, tool-calling failures, or "doom loops" where models get stuck in repetitive outputs.
• Ternary quantization, which uses -1, 0, and 1, is being distinguished from pure binary 1-bit methods, as the former can achieve higher accuracy by utilizing specific scaling factors across weight groups.
• The technical implementation of these models requires specialized software support, as standard tools like LM Studio or vanilla llama.cpp may not yet fully support the unique architectures or custom kernels required for these highly compressed formats.
• Some observers express skepticism toward new quantization benchmarks, noting that reported results often rely on specific evaluation suites that may not reflect real-world performance or "vibe" consistency compared to original, less-compressed models.
• There is significant curiosity about the business strategies of small AI labs, with speculation that funding from major manufacturers like Samsung points toward a goal of deploying capable on-device AI to compete with Apple's integration.
• Distinctions are being made between general-purpose LLMs and specialized small-scale models, with some suggesting that for certain mobile tasks, a smaller, fine-tuned model is superior to a heavily compressed, larger general-purpose model.
• The community is actively building personal evaluation pipelines to verify performance claims, reflecting a lack of trust in marketing materials and a desire for standardized metrics across different quantization regimes.
• The mention of potential talks between developers and major tech companies like Apple has sparked discussion on the tension between corporate secrecy and the open-source nature of current AI research.
• Concerns were raised regarding the quality of communication, with some finding the technical writing in recent AI announcements to be overly "marketing-speak" or potentially AI-generated, which distracts from the underlying technical achievements.
The shift toward extreme quantization represents a significant technical effort to fit large-scale intelligence into constrained environments like mobile devices or consumer laptops. While the potential to run high-parameter models locally is compelling, significant friction remains regarding software compatibility, model reliability, and the trade-offs between size and reasoning capability. Consensus suggests that while these methods are a promising direction for edge computing, they currently struggle with stability issues that may hinder their immediate use as daily-driver assistants. Overall, the community is moving toward a more critical, data-driven approach to evaluating these models, favoring rigorous internal benchmarks over the initial, often optimistic, claims provided by new research labs.