OpenArch – PyTorch implementations of modern LLM architectures
OpenArch 是一个提供现代开源大规模语言模型架构手写 PyTorch 实现的仓库。通过从零构建这些模型，项目定位为教育资源，而非可替代用于生产的库（如 transformers）。该仓库强调将注意力机制、归一化技术和混合专家路由等结构选择明确化，便于在不同模型间进行比较。
现代语言模型有共同的结构基础，但在具体设计上差异很大，包含多种注意力形式（如 Multi-Head 、 Grouped-Query 、 Latent Attention）以及多种归一化方法（如 RMSNorm 、 QK-Norm）。项目把每种架构独立为单个、可读性强的文件，让开发者在不被为追求速度和 sharding 而产生的生产优化复杂性干扰的情况下，研究这些细节。
仓库的实现列表在不断增长，涵盖从 GPT-2 这样的早期模型到 DeepSeek R1 、 Llama 4 Maverick 、 Kimi K2 等大型复杂架构。项目通过状态表跟踪进度，标注模型是已能完成前向计算（forward pass）还是仍在开发中。除了纯文本模型，集合也在扩展到多模态架构，像 PaliGemma 的相关工作已在进行中。
该项目以协作学习为导向，鼓励有兴趣分析论文和模型配置文件的人参与贡献。作者希望将仓库打造成社区驱动的平台，用以记录并解读现代 AI 架构的多样性。欢迎贡献者通过添加新模型实现、改进文档或创建用于与官方模型权重对照验证性能的测试用例，共同扩展该库。
该工作大量依赖 Sebastian Raschka 和 Machine Learning Mastery 团队等专家的教学资料，他们的成果是这些实现所依据的架构图与比较的主要参考。项目旨在成为有价值的学习工具，但作者强调它与原始模型创建者无关，且不适合用于生产环境。
OpenArch is a repository dedicated to providing hand-written PyTorch implementations of modern open-source large language model architectures. By building these models from scratch, the project aims to serve as an educational resource rather than a replacement for production-ready libraries like transformers. The repository focuses on making structural choices, such as attention mechanisms, normalization techniques, and mixture-of-experts routing, explicit and easy to compare across different models.
Modern language models share a common structural foundation but utilize a wide variety of specific design configurations. These include different types of attention mechanisms like Multi-Head, Grouped-Query, and Latent Attention, alongside diverse normalization methods such as RMSNorm and QK-Norm. By isolating each architecture into a single, readable file, the project allows developers to study these nuances without the complexity often found in production-optimized codebases, which prioritize speed and sharding over conceptual clarity.
The repository currently features a growing list of implementations, ranging from early models like GPT-2 to massive, complex architectures like DeepSeek R1, Llama 4 Maverick, and Kimi K2. The project tracks its progress through a status table that indicates whether a model is fully functional for forward passes or currently under construction. Beyond pure text models, the collection also aims to encompass multimodal architectures, with work already underway on models such as PaliGemma.
Designed with a focus on collaborative learning, the project encourages contributions from those interested in analyzing research papers and model configuration files. The author envisions the repository as a community-driven effort to document and demystify the architectural diversity of modern AI. Contributors are invited to help expand the library by adding new model implementations, improving documentation, or creating test cases that verify performance against official model weights.
This effort relies heavily on educational materials from experts such as Sebastian Raschka and the team at Machine Learning Mastery. Their work serves as the primary reference for the architectural diagrams and comparisons that guide these implementations. While the project is meant to be a valuable tool for learning, the author emphasizes that it is not affiliated with the original model creators and is not intended for production environments.
29 comments • Comments Link
在 PyTorch 中从头实现现代 LLM 架构，是理解那些被高级库掩盖的设计权衡和细节的有力教学手段。
推理生态从大多数模型都遵循近乎统一的 GPT 式架构，演变为多样化设计的"寒武纪大爆发"——出现了定制的 attention 机制、线性 attention 以及稀疏 Mixture-of-Experts (MoE) 层等。
推理引擎的实现并非现成可商品化的任务。供应商通常需构建或大幅改造执行栈（如 vLLM 、 TensorRT-LLM 或专有方案），以满足 tensor parallelism 、高效 MoE routing 等架构需求。
LLM 的概率本质使得验证推理实现的正确性变得困难：细微优化或硬件级近似可能会以在没有严格基准测试下难以发现的方式改变模型表现。
为快速变化的 open-weight 模型生态开发并维护后端非常耗费资源，通常只有大型企业才能承担的专业工程团队才能完成。
与 Hugging Face 的 `transformers` 等成熟库交叉比对自定义实现，仍是验证架构代码正确性的主要手段。
像 `llama.cpp` 这样受欢迎的仓库表明，高水平的社区参与和技术专长对于跟进现代量化技术与专用模型变体的复杂性至关重要。
模型架构（在 PyTorch 中定义的"脑"或"代理"结构）与训练流水线（负责数据输入、损失函数和权重更新的训练环境或 RL 环境）之间存在明显差别。
用纯 NumPy 而非 PyTorch 实现模型，有时能带来更深入的教学体验，剥离框架抽象以暴露底层矩阵运算，但代价是运行速度较慢。
鉴于行业的快速演进，维护个人的模型实现集合非常具有挑战性，这凸显了社区驱动贡献对项目长期可持续性的关键作用。
LLM 架构多样化的迅速扩张，使行业从过去"一刀切"的推理运行时走向碎片化的格局，越来越依赖对 attention 机制、 MoE routing 以及并行化等方面的专业知识。尽管从零构建这些架构能深入理解论文中的技术细节，但也暴露了学术定义与大规模部署所需工程实践之间的脱节。随着模型愈发复杂，个人开发者难以跟上并实现每一种新架构变体，因而协作性的开源努力对于维护准确且高效的参考实现变得不可或缺。 • Implementing modern LLM architectures from scratch in PyTorch serves as a powerful pedagogical tool for internalizing design choices that are otherwise obscured by high-level libraries.
• The inference ecosystem has evolved from a state where most models shared a near-uniform GPT-style architecture to a "Cambrian explosion" of diverse designs, including custom attention mechanisms, linear attention, and sparse Mixture-of-Experts (MoE) layers.
• Inference engine implementation is not a commodity task. Providers must often build or heavily modify execution stacks—such as vLLM, TensorRT-LLM, or proprietary solutions—to handle specific architectural requirements like tensor parallelism and efficient MoE routing.
• The probabilistic nature of LLMs makes verifying the correctness of an inference implementation difficult, as subtle optimizations or hardware-level approximations can alter model performance in ways that are hard to detect without rigorous benchmarks.
• Developing and maintaining a backend compatible with the rapidly shifting landscape of open-weight models is resource-intensive, often requiring the specialized engineering teams that only large-scale industry players can afford.
• Cross-referencing custom implementations against established repositories like Hugging Face's `transformers` remains the primary method for ensuring the accuracy of architectural code.
• Popular repositories like `llama.cpp` demonstrate that high levels of community engagement and technical expertise are required to keep up with the complexity of modern quantization techniques and specialized model variants.
• There is a clear distinction between the model architecture (the "brain" or "agent" structure defined in PyTorch) and the training pipeline (the "gym" or RL environment that governs data input, objective functions, and weight updates).
• Implementing models in pure NumPy rather than PyTorch can sometimes offer a more profound pedagogical experience, stripping away framework abstractions to expose the underlying matrix operations at the cost of execution speed.
• Maintaining an individual collection of model implementations is a significant challenge given the industry's rapid pace, suggesting that community-driven contributions are essential for long-term project viability.
The rapid proliferation of diverse LLM architectures has shifted the industry away from a "one-size-fits-all" inference runtime toward a fragmented landscape where specialized knowledge of attention mechanisms, MoE routing, and parallelization is required. While building these architectures from scratch provides deep insight into technical papers, it also highlights the disconnect between academic definitions and the practical engineering required to deploy models at scale. As models grow increasingly complex, the barrier to entry for individual developers to track and implement every new architectural variant is rising, necessitating collaborative open-source efforts to maintain accurate and performant references.