Baidu released ERNIE 5.5 Flash Base
187 points
• 7 days ago
• Article
Link
ERNIE 5.5 Flash Base 是一款开放权重的基座模型，旨在为领域适应、推理和 Agentic 应用等方向的后续训练提供高效的起点。该模型总参数量为 2800 亿，每个 token 激活 140 亿个参数，采用稀疏的 Mixture-of-Experts 架构。它在 33.5 万亿个 token 上从零开始预训练：预训练阶段支持 128,000 个 token 的上下文窗口，微调与推理阶段最高可达 100 万个 token 。通过以原始预训练检查点的形式发布模型，开发团队希望为研究者和开发者提供更大的灵活性，让他们能够自行选择后续的训练策略。
效率是该模型设计的核心，贯穿每一个架构决策。架构采用混合注意力机制：大多数层使用 Heavily Compressed Attention 以降低内存与计算开销，而部分 Multi-head Latent Attention 层则用于处理细粒度的远程依赖。为进一步提升性能，模型集成了 Multi-Router Latent MoE 以优化专家选择，采用 Manifold-Constrained Hyper-Connections 改善信息流，并配置 MultiMax 输出头，可动态调整温度，从而最大限度减小不合逻辑 token 的影响。此外，ECHO 模块通过训练骨干网络直接进行前瞻预测，替代传统的多 token 预测方式，从而增强推理能力。
预训练数据来自多样化的数据集，重点覆盖高质量的网络内容、仓库规模的代码以及长上下文文档。为应对大规模训练，团队开发了专门的基础设施，包括分布式 Muon 优化器以及 FlashMask-4 注意力内核。这些工具与模型架构协同设计，用于优化数千块 GPU 上的通信模式与内存占用，确保在模型规模与复杂度都很高的情况下仍能高效训练。生成的模型使用新的 Right-to-left tokenizer 以提升数值准确度，与同等规模的其他开源模型相比，在学术、推理和代码相关基准测试中展现出较强表现。
ERNIE 5.5 Flash Base 目前已在 Hugging Face 提供 FP8 与 BF16 两种格式，便于开发者借助 vLLM 或 SGLang 等标准工具将其接入现有流程。本次发布定位为基础版本，开发团队也提到未来计划推出该模型的后训练版本。通过共享这些基础权重，团队为社区提供了一个可控的起点，让大家在不必承担大规模预训练的高昂成本的前提下，探索新的训练方法与 Agentic 工作流。
ERNIE 5.5 Flash Base is an open-weights base model designed to provide a highly efficient starting point for further training in areas like domain adaptation, reasoning, and agentic applications. The model features 280 billion total parameters with 14 billion activated per token, using a sparse Mixture-of-Experts architecture. It was pre-trained from scratch on 33.5 trillion tokens, supporting a context window of 128,000 tokens during pre-training and up to one million tokens for fine-tuning and inference. By releasing the model as a raw pre-training checkpoint, the developers aim to give researchers and developers the flexibility to pursue their own post-training strategies.
Efficiency serves as the cornerstone of the model's design, influencing every architectural decision. The architecture utilizes a hybrid attention system where most layers employ Heavily Compressed Attention to reduce memory and compute costs, while a selection of Multi-head Latent Attention layers handles detailed, long-range dependencies. To further optimize performance, the model incorporates Multi-Router Latent MoE to improve expert selection, Manifold-Constrained Hyper-Connections for better information flow, and a MultiMax output head that dynamically adjusts temperature to minimize the impact of implausible tokens. Additionally, the ECHO module replaces traditional multi-token prediction by training the backbone to look ahead directly, which enhances reasoning capabilities.
The pre-training process relied on a diverse dataset emphasizing high-quality web content, repository-scale code, and long-context documents. To manage training at scale, the team developed specialized infrastructure including a distributed Muon optimizer and the FlashMask-4 attention kernel. These tools were co-designed with the model architecture to optimize communication patterns and memory usage across thousands of GPUs, ensuring that the model could be trained efficiently despite its size and complexity. The resulting model, using a new right-to-left tokenizer for numerical accuracy, shows strong performance across academic, reasoning, and coding benchmarks when compared to other open-source models of similar scale.
ERNIE 5.5 Flash Base is currently available on Hugging Face in both FP8 and BF16 formats, allowing developers to integrate it into their existing pipelines using standard tools like vLLM or SGLang. The release is intended as a foundation, with the developers noting that a post-trained version of the model is planned for the future. By sharing these base weights, the team provides a controlled baseline for the community to experiment with new training methods and agentic workflows without the prohibitive cost of large-scale pre-training.
97 comments • Comments Link
• 该新模型的许可协议十分严格，明确禁止在 EU 、 UK 和 South Korea 使用，并将合规责任归于最终用户而非发布者。
• 这种地域限制通常被解读为一种具有战略意味的"回避条款"，目的在于让开发者免于卷入与 EU AI Act 和 GDPR 相关的法律复杂性及监管成本，而并非真正试图落实本地限制。
• 虽然该许可在实践中未必能够有效执行，但它起到了法律层面的责任转移作用，明确传递出开发者不对该模型在受监管地区的部署承担责任。
• 发布 base models 被认为是生态系统的积极趋势：这些模型为开发者提供了可用于自主微调与对齐的基础，从而避免专有且预对齐模型那种"黑箱式"的限制。
• 对训练资源的投入如何分配存在争议：有人认为 base training 占据了绝大多数工作量，也有人认为现代 post-training（RLHF/RLVR）在总计算量与复杂度中同样占据关键比重。
• base models 的去中心化发布被视为可能阻止 AI 权力过度集中的一项保障机制，因为它更鼓励多元化的本地对齐工作，或许能对冲大型企业主导的 frontier models 的影响。
• 技术讨论显示，该模型采用一种高效的 14B active parameter 的 Mixture-of-Experts（MoE）架构；在用户能够完成必要的 post-training 与对齐步骤的前提下，它有望较合理地适配高端本地硬件。
• 行业观察人士认为，这一版本的发布整体上与其他近期的 base models 处于同一竞争区间，尽管由于迭代速度过快以及不同模型版本之间存在较大架构差异，对比变得更复杂。
• 命名方式尤其是对 "ERNIE" 的使用，唤起了对 BERT 以及 Sesame Street 的怀旧联想；而用 "dropped" 来描述发布，则反映出科技行业在游戏和零售文化影响下不断演化的俚语用法。
• 对 base models 用于非专家人群的实际价值，仍有不少质疑：这些模型要满足指令要求或对特定下游任务真正有用，通常需要大量技术资源与专业知识。
这场讨论凸显了两类人群之间的分歧：一方认为发布 base model 是通向民主化的关键一步；另一方则对潜在商业意图或此类模型对普通用户的可行性抱有怀疑。限制性许可被普遍视为一种偏防御性质的法律工具，而非技术层面的必需品，这进一步反映出 AI 快速发展与 EU 等地区复杂且分散的监管环境之间的摩擦正在加剧。尽管专家们认可 base models 在本地部署方面的效率与透明度，但越来越清楚的是，安全、对齐以及法律合规的成本与责任，正在被更大程度地转移到那些需要把模型整合进自身工作流的从业者与企业身上。总体而言，这次发布也体现了 frontier developers 管理全球风险方式的变化：他们更倾向于提供更广泛的可用性，并附带"撇清责任"的条款，而不是在此前更受控的发布节奏中那种死板且严格的访问限制。 • The license for the new model is notably restrictive, explicitly forbidding use in the EU, UK, and South Korea, and placing the burden of compliance on the end-user rather than the publisher.
• The geographic exclusion is widely interpreted as a strategic "escape clause" designed to shield the developer from the legal complexities and regulatory costs associated with the EU AI Act and GDPR, rather than a genuine attempt to enforce local restrictions.
• While the license is likely unenforceable in practice, it functions as a legal mechanism to offload liability, effectively signaling that the developer accepts no responsibility for how the model is deployed in regulated territories.
• Releasing base models is viewed as a positive trend for the ecosystem, as these models provide the foundation for developers to perform their own fine-tuning and alignment, avoiding the "black box" constraints of proprietary, pre-aligned models.
• There is a debate regarding the resource distribution of training, with some arguing that base training is the overwhelming majority of the effort, while others suggest that modern post-training (RLHF/RLVR) constitutes a significant portion of total compute and complexity.
• The decentralized release of base models is presented as a potential safeguard against the centralization of AI power, as it encourages diverse, local alignment efforts that may balance out the influence of large, corporate-controlled frontier models.
• Technical discussions suggest the model is an efficient 14B active parameter Mixture-of-Experts (MoE) architecture that fits reasonably within high-end local hardware, provided the user handles the necessary post-training and alignment steps.
• Industry observers note that the release appears competitive with other recent base models, though comparisons are complicated by the rapid pace of iteration and the distinct architectural differences between model versions.
• The nomenclature, specifically the use of "ERNIE," evokes nostalgic comparisons to BERT and Sesame Street, while the use of "dropped" to describe the release reflects evolving tech-industry slang influenced by gaming and retail cultures.
• Skepticism persists regarding the utility of "base" models for non-experts, as they require significant technical resources and expertise to make them instruction-compliant or useful for specific downstream tasks.
The conversation reflects a divide between those who view base model releases as a critical step toward democratization and those who remain skeptical of the underlying business intent or the feasibility of such models for the average user. The restrictive licensing is widely recognized as a defensive legal maneuver rather than a technical necessity, highlighting the increasing friction between rapid AI development and the complex, fragmented regulatory landscape in regions like the EU. While experts appreciate the efficiency and transparency of base models for local deployment, there is a clear acknowledgment that the burden of safety, alignment, and legal compliance now shifts heavily onto the practitioners and businesses integrating these models into their workflows. Overall, the release underscores a shift in how frontier developers are managing global risk, favoring broad availability with "wash-your-hands" clauses over the rigid, restrictive access seen in previous, more controlled release cycles.