Running Gemma 4 26B at 5 tokens/sec on a 13-year-old Xeon with no GPU
326 points
• 1 day ago
• Article
Link
一台服役已十三年的 HP StoreVirtual 服务器，配备两颗 Ivy Bridge Xeon E5‑2690 v2 处理器且无 GPU，已成功运行 Google 的 Gemma 4 26B‑A4B 语言模型。尽管缺少 AVX2 、 FMA3 等常见现代指令集，它仍能以约每秒 5 个 token 的速度生成文本。该项目实际证明，现代 AI 能力可以在遗留企业硬件上得到利用，挑战了"高性能 AI 只能靠云订阅或昂贵现代基础设施"的观念。
最初团队试图复现另一篇关于在旧硬件上运行语言模型的文章中的流程，但首次尝试失败：用于 Mixture‑of‑Experts 优化的专用 ik_llama.cpp 分支依赖于 Intel 在 2014 年 Haswell 架构中才支持的 AVX2 指令。借助 AI 助手分析构建失败并排查代码库，团队找到了必要的修改，使软件在遇到缺失指令时能够回退到标量计算。
技术上的关键在于修复模型计算调度器中的一个"静默失败"。图构建器生成了 Ivy Bridge 上没有对应计算路径的操作，导致模型产出看似流畅但毫无意义的文本。通过用可移植的标量循环重写缺失操作，并确保图构建器在禁用特定优化标志时能正确处理，模型才得以正常运行。这个过程凸显了现代工程的一项关键能力：审计并改造性能关键代码以适配具体硬件约束。
除了具体的技术成就外，该实验还强调了实操经验在维护遗留系统时的重要性。成功并不在于从头重写整个引擎，而在于系统性地诊断故障并实施有针对性的补丁。关闭诸如为新指令集设计的运行时权重重新打包等选项后，这台服务器现在可作为 AI 任务的可行本地回退方案，表明爱好者和企业有可能在本应报废的旧硬件上维持高效的本地 AI 能力。
A repurposed, thirteen-year-old HP StoreVirtual server, equipped with two Ivy Bridge Xeon E5-2690 v2 processors and no GPU, has been successfully configured to run Google's Gemma 4 26B-A4B language model. While this hardware lacks the modern instruction sets, such as AVX2 and FMA3, typically required for such tasks, it now generates text at a respectable five tokens per second. This project serves as a practical demonstration that modern artificial intelligence capabilities can be harnessed on legacy enterprise hardware, challenging the assumption that high-performance AI is exclusively accessible through cloud subscriptions and expensive, modern infrastructure.
The project began as an attempt to replicate a workflow described in another article regarding running language models on older hardware. However, the initial attempt failed because the specialized ik_llama.cpp fork, which includes optimizations for Mixture-of-Experts models, relied on AVX2 instructions that were not introduced until Intel's Haswell generation in 2014. By utilizing an AI assistant to analyze the build failures and troubleshoot the codebase, the necessary modifications were identified to allow the software to fall back on scalar math instead of reaching for non-existent CPU instructions.
The technical core of the fix involved addressing a silent failure within the model's compute dispatcher. Because the graph builder was emitting operations for which no compute path existed on the Ivy Bridge architecture, the model produced fluent-looking but entirely nonsensical output. By re-implementing the missing operations using portable scalar loops and ensuring the graph builder correctly handles cases where specialized optimization flags are disabled, the model was finally able to function. This process highlights a critical skill for modern engineering, namely the ability to audit existing, performance-critical code and make it compatible with specific hardware constraints.
Beyond the specific technical achievement, this experiment underscores the value of hands-on expertise in navigating legacy systems. The success of the project did not require rewriting the entire engine from scratch, but rather systematically diagnosing why the system was failing and implementing targeted patches. By turning off flags like runtime weight repacking, which is designed for newer instruction sets, the server now operates as a viable local fallback for AI tasks. This highlights the potential for enthusiasts and businesses alike to maintain high-functioning, local AI capabilities on older hardware that might otherwise be discarded.
208 comments • Comments Link
• 预测显示，到 2027 年中期，大规模 Mixture-of-Expert 模型将在消费级硬件上变得可行，一些行业专家预计在参数压缩和三值量化方面会更快取得突破。
• 目前在 MacBook 等消费设备上的本地运行已达到可用的速度，但对于每秒 7–9 个 token 是算"还行"还是仅适合非实时工作流，各方仍有争议。
• 硬件利用策略在不断演化以平衡散热与能效，包括限制核心使用或将任务调度到空闲时段等手段。
• 本地推理相较云端服务的经济可行性，很大程度取决于当地电价、硬件效率和对数据隐私的需求；在特定条件下，一些用户发现本地执行更有成本优势。
• 在老旧硬件上进行大模型推理时，内存带宽是主要瓶颈，通常需要在量化级别（例如 Q4 与 Q8）之间权衡以兼顾精度和速度。
• 实际应用更看重上下文窗口大小和输出质量，而不是单纯的 token 生成速率，尤其是在代码审查或复杂的 agent 式自动化等任务中。
• 未来的进展预计会超越单纯的参数数量，更多聚焦于更高效的架构，例如递归网络或高级噪声函数权重生成技术，以在不牺牲性能的情况下减少内存占用。
• 使用老旧硬件（例如 2013 年的 Xeon 服务器）仍是本地 LLM 实验的可行途径，但通常需要自定义补丁来弥补架构限制，例如不支持 AVX2 指令。
• 在追求高速交互式 AI 性能的用户与采用"delegate"委派工作流的用户之间存在明显分歧，后者将 AI 任务排队并异步处理，类似于 3D 打印作业的模式。
• 关于 AI 生成内容的担忧正在浮现，用户争论依赖 LLMs 编写技术文章或补丁是否会削弱社区的真实性与可信度。
此次讨论反映了一个过渡期：在本地运行前沿级模型正从理论上的不可能，逐步变为一种虽然资源密集但可行的现实。尽管高端 GPU 集群目前仍主导速度竞赛，社区中出现了一批致力于最大化现有消费级硬件效率的用户群体。关于这些设置是否"有用"尚无共识，关键在于用户是把模型当作交互式助手，还是视为异步的后台工具。归根结底，这场讨论既彰显了技术快速民主化带来的兴奋，也暴露了内存带宽、散热管理与硬件架构等现实限制之间的紧张关系。 • Projections suggest that by mid-2027, large-scale Mixture-of-Expert models will be viable on consumer hardware, with some industry experts anticipating even faster breakthroughs in parameter compression and ternary quantization.
• Current local performance on consumer devices like MacBooks allows for functional speeds, though debates persist regarding whether 7–9 tokens per second is "decent" or merely a curiosity for non-real-time workflows.
• Hardware utilization strategies are evolving to balance thermal output and energy efficiency, including techniques like limiting core usage or scheduling tasks to run during idle periods.
• The economic viability of local inference compared to cloud-based providers depends heavily on local electricity costs, hardware efficiency, and the necessity of data privacy, with some users finding local execution cost-effective under specific conditions.
• Memory bandwidth is the primary bottleneck for large-scale model inference on older hardware, often necessitating trade-offs between quantization levels (e.g., Q4 vs. Q8) to balance precision and speed.
• Real-world applications often prioritize context window size and output quality over raw token generation speed, particularly for tasks like code review or complex agentic automation.
• Future progress is expected to move beyond simple parameter count, focusing on more efficient architectures like recursive networks or advanced noise-function weight generation that reduce the memory footprint without sacrificing performance.
• The use of aging hardware—such as 2013-era Xeon servers—is a viable pathway for local LLM experimentation, though it often requires custom patches to address architecture-specific limitations like missing AVX2 instructions.
• There is a clear divide between users who demand high-speed, interactive AI performance and those who utilize "delegate" workflows, where AI tasks are queued and processed asynchronously similar to 3D printing jobs.
• Concerns regarding AI-generated content in discussions are surfacing, as users debate whether relying on LLMs to write technical posts or patches undermines the authenticity and credibility of the community.
The discussion reflects a transition period where running frontier-class models locally is shifting from a theoretical impossibility to a practical, albeit resource-intensive, reality. While high-end GPU farms currently dominate the speed race, there is a burgeoning segment of the community focused on maximizing efficiency on existing consumer hardware. Consensus on the "usefulness" of these setups remains elusive, as it depends on whether the user views the model as an interactive assistant or an asynchronous background utility. Ultimately, the conversation highlights a tension between the excitement of rapid technological democratization and the practical constraints of memory bandwidth, thermal management, and hardware architecture.