Mesh LLM: distributed AI computing on iroh
347 points
• 5 days ago
• Article
Link
现代大型语言模型通常被锁定在昂贵的集中式数据中心，用户因此失去对硬件、数据隐私以及不断变化的模型策略的控制。 Mesh LLM 挑战了这一现状：它将分布式机器上的现有 GPU 和内存资源汇聚整合为一个统一的整体，使用户能够在本地运行模型、与团队共享算力，甚至把大型模型拆分到多台设备上协同运行，同时通过熟悉的 OpenAI-compatible API 进行交互。
该架构依赖一个可插拔的运行时来管理各种模型，从小型轻量级选项到数十亿参数的超大模型。为了解决单机容量不足的问题，Mesh LLM 采用名为 Skippy 的流水线分区方法，将模型按层范围拆分，使多台普通机器能够协作执行同一模型，最终用户无需手动处理复杂的分布式细节。
系统的核心是 iroh——一个负责经身份验证的点对点连接的网络库。 Mesh LLM 的每个节点都作为 iroh 端点运行，借助 QUIC 实现稳健且安全的通信，从而无需中央服务器。通过 NAT traversal 和 hole-punching，iroh 让节点无论地理位置如何都能直接互联，从而把网络层有效地抽象掉，使得路由到远端对等节点的体验如同与 localhost 通信般无缝。
该协议基于 QUIC 的 ALPN 协商，为 gossip 、 control 和对延迟敏感的 activation transport 提供独立数据流。连接通过公钥进行身份验证，并由自定义的 gossip 层管理，因此系统对安全性和节点信任保持严格控制。该框架支持 Mesh LLM 的有机扩展，用户可以随时向其网格添加节点，无论是在私有部署中还是参与更广泛的公共网络。
展望未来，项目计划通过支持诸如 ACP 等新兴标准，进一步减少对封闭服务器环境的依赖。凭借轻量的软件体量和对开放点对点计算的承诺，Mesh LLM 为开发者重新掌控 AI 基础设施提供了切实可行的路径。通过把闲置硬件变为统一的高性能网络，它为当今普遍存在的计量式黑盒模型提供了一种可扩展的替代方案。
Modern large language models are typically tethered to expensive, centralized data centers where users lose control over hardware, data privacy, and evolving model policies. Mesh LLM challenges this status quo by pooling existing GPU and memory resources across distributed machines, transforming them into a single, cohesive unit. This approach enables users to run models locally, share compute power with teams, or even split massive models across several devices, all while interacting through a familiar, OpenAI-compatible API.
The architecture relies on a pluggable runtime that manages various models, ranging from small, lightweight options to massive, multi-billion-parameter giants. To handle models that exceed the capacity of a single machine, Mesh LLM employs a pipeline-based partitioning method known as Skippy. This system splits the model into layer ranges, allowing multiple modest machines to collaboratively execute a single model without the end user needing to manage the complex distribution process.
At the core of this system is iroh, a networking library that handles authenticated, peer-to-peer connections. Each node in the mesh operates as an iroh endpoint, utilizing QUIC for robust, secure communication that bypasses the need for a central server. By leveraging NAT traversal and hole-punching, iroh allows nodes to communicate directly regardless of their physical location. This ensures that the networking layer is effectively abstracted away, making routing to a remote peer feel as seamless as communicating with localhost.
The protocol runs on QUIC's ALPN negotiation, which facilitates different streams for gossip, control, and latency-sensitive activation transport. Because these connections are authenticated by public keys and managed by a custom gossip layer, the system maintains strict control over security and peer trust. This framework allows Mesh LLM to scale organically, letting users add nodes to their mesh at will, whether they are running private deployments or participating in the broader public network.
Looking ahead, the project aims to further reduce dependency on closed server environments by supporting emerging standards like ACP. With a lightweight software footprint and a commitment to open, peer-to-peer computing, Mesh LLM provides a practical path for developers to regain control over their AI infrastructure. By turning idle hardware into a unified, high-performance network, it offers a scalable alternative to the metered, black-box model prevalent in the industry today.
94 comments • Comments Link
- 在消费级硬件上，分布式模型推理正在受到关注；演示显示在多个节点间可达到约每秒 10–16 个 token 的吞吐率。
- 目前的方法通过专用引擎将大型模型切分到不同机器上，利用 1 Gbit Ethernet 等标准网络协议，无需依赖专用的 RDMA 或 NVLINK fabric 。
- 性能受制于瓶颈：在自回归解码过程中，每生成一个 token 需要传输约 2 × hidden size × 分片数量 的数据量。
- 量化策略（例如对敏感 tensors 保持 Q8，而对其他部分进行激进压缩）对于在受限硬件上控制每层计算时间至关重要。
- 框架采用分阶段流水线架构，每个节点维护自己的 key-value cache，并通过跨阶段同时处理多个查询来实现高并行性。
- 系统通过动态路由内置可靠性，支持拓扑重算，并在节点在推理过程中掉线时自动重试请求。
- 隐私仍然是重大挑战：分布式计算意味着中间节点以明文处理 model activations，对于敏感任务需要使用私有且受信任的 mesh 。
- 传输安全通过加密的 QUIC 得到保障，确保尽管计算本身以明文进行，但节点间传输的数据是受保护的。
- 开发者正积极将 token prediction 和 n-gram speculative decoding 等技术集成进来，以进一步提升吞吐量并更好利用并行性。
- 该项目作为专有分布式计算解决方案的替代方案，专注于实现点对点的模型共享，而无需企业级专用硬件。
此次讨论突显了人们通过利用分布式消费级硬件来实现大型语言模型推理民主化的强烈技术热情。尽管在隐私保护和标准网络协议固有的带宽 / 延迟限制方面仍存在重大障碍，但将模型切分到异构节点的能力代表了向去中心化 AI 的重要转变。参与者对技术架构感到兴奋，同时也对明文计算和性能下降等现实问题保持谨慎。前提是用户能够为其计算任务构建可靠且受信任的点对点网络，这种方法可视为集中式云服务的有力替代。 • Distributed model inference on consumer hardware is gaining traction, with demonstrations achieving approximately 10–16 tokens per second across multiple nodes.
• The current approach uses a specialized engine to split large models across separate machines, utilizing standard networking protocols like 1Gbit Ethernet without requiring dedicated RDMA or NVLINK fabric.
• Performance is bottlenecked by the requirement to transfer approximately 2 multiplied by the hidden size and the number of shards per token during autoregressive decoding.
• Quantization strategies, such as preserving sensitive tensors at Q8 while aggressively compressing others, are critical to managing the compute time per layer on limited hardware.
• The framework employs a staged pipeline architecture where each node manages its own key-value cache, allowing for high parallelism by processing multiple queries simultaneously across stages.
• Reliability is built into the system through dynamic routing, allowing the topology to recalculate and automatically retry requests if a node drops out mid-inference.
• Privacy remains a significant challenge, as distributing compute means intermediate nodes process model activations in cleartext, necessitating the use of private, trusted meshes for sensitive tasks.
• Transport security is handled via encrypted QUIC, ensuring that while the computations themselves are cleartext, the data in transit between nodes is protected.
• Development is actively addressing the integration of token prediction techniques and ngram speculative decoding to further optimize throughput and utilize available parallelism.
• The project serves as an alternative to proprietary distributed compute solutions, focusing on enabling peer-to-peer model sharing without the need for specialized enterprise hardware.
The discussion highlights a growing technical enthusiasm for democratizing large language model inference by leveraging distributed consumer hardware. While significant hurdles remain regarding privacy and the inherent speed limitations of standard networking protocols, the ability to split models across heterogeneous nodes represents a major shift toward decentralized AI. Participants expressed both excitement for the technical architecture and caution regarding the practical realities of cleartext computation and performance degradation. Ultimately, this approach is seen as a compelling alternative to centralized cloud services, provided users can curate reliable, trusted peer networks for their compute tasks.