Amazon saves millions by replacing Sonnet 4.6 with Strands Decider
725 points
• 2 days ago
• Article
Link
亚马逊工程师最近通过用一种名为 Strands Decider 的专用决策模型取代 Claude Sonnet 4.6 大语言模型，为其内部票务系统实现了显著的成本节约。尽管 Sonnet 4.6 表现不错，但切换到 Strands Decider 后，运营成本下降了 98%，公司有望每年节省数百万美元。这次变化反映出一种战略转向：在高频、结构化任务中，优先使用更精简、为特定场景打造的模型。
该票务系统需要处理大量输入，包括客户升级诉求、安全发现以及容量请求，并将它们归入 20 个不同的类别。与生成对话文本的传统 LLM 不同，决策模型会接收输入状态，并对预定义选项集合返回概率分数。借助 Strands Decider E4B 模型，团队成功用更高效的决策流程替代了冗长的基于 JSON 的 LLM 提示词，从而显著减少了所需的 Token 数量。
为确保平稳切换，团队通过 AWS AppConfig 实施了三向配置开关，能够实时对比两种模型效果。他们采用双模式监控流量：在新系统全面上线前，Sonnet 和 Decider 并行运行以验证性能。该方式比离线评估可靠得多，因为它让团队能够基于真实生产规模数据分析差异。
此次切换的效果十分突出：每个工单的平均延迟下降了 78% 。通过优化输入提示词，并将输出精简为单一选择而不是完整的 JSON 响应，每个工单的 Token 总数减少了 70% 。此外，团队还发现：在请求中对 JSON 键进行排序能够提升一致性，他们注意到键的顺序此前会影响模型输出的概率。
尽管结果令人满意，工程师们仍强调决策模型更适用于特定场景，例如类别集合稳定的高并发分类任务。他们建议，如果某项任务需要较长解释，或类别目录经常变化，传统 LLM 可能仍然更合适。最终，这次迁移的成功表明：在大型企业环境中为代理部署专用决策模型，能够带来可观的经济收益。
Amazon engineers recently achieved significant cost savings by replacing the Claude Sonnet 4.6 large language model with a specialized decision model known as Strands Decider for their internal ticketing system. While Sonnet 4.6 performed effectively, the transition to Strands Decider reduced operational costs by 98 percent, potentially saving the company millions of dollars annually. The change highlights a strategic shift toward using leaner, purpose-built models for high-volume, structured tasks.
The ticketing system handles a vast array of inputs, including customer escalations, security findings, and capacity requests, which must be sorted into 20 distinct categories. Unlike traditional LLMs that generate conversational text, a decision model accepts input state and returns probability scores for a predefined set of options. By leveraging the Strands Decider E4B model, the team successfully replaced lengthy JSON-based LLM prompts with a more efficient decision-making process that requires significantly fewer tokens.
To ensure a safe transition, the team implemented a three-way configuration switch through AWS AppConfig, allowing them to compare models in real-time. They monitored traffic in a dual-mode, running both Sonnet and the Decider in parallel to validate performance before fully committing to the new system. This method proved far more reliable than offline evaluations, as it allowed the team to analyze discrepancies in live, production-scale data.
The performance metrics of this switch were notable, with average latency per ticket dropping by 78 percent. By optimizing the input prompts and streamlining the output to a single selection rather than a full JSON response, the total token count per ticket decreased by 70 percent. Additionally, the team discovered that sorting JSON keys during requests improved consistency, noting that key order had previously influenced model output probabilities.
While the results were impressive, the engineers emphasized that decision models are best suited for specific use cases, such as high-volume classification tasks with a stable set of categories. They advised that if a task requires lengthy explanations or frequent changes to the category catalog, traditional LLMs might still be more appropriate. Ultimately, the success of this migration demonstrates the substantial economic benefits of deploying specialized decision models for agents within large-scale enterprise environments.
261 comments • Comments Link
- 行业经历了周期性的转向：一开始为构建高效、专用的 machine learning classifiers 所做的努力，被强大的 LLMs 出现所打断；但当各组织在生产环境中寻求更具成本效益、可实现高吞吐量的方案时，市场又重新回到轻量级 classifiers 。
- LLMs 通过打破"需要海量、人工策划数据集"这一曾经让模型很难落地的高门槛，彻底改变了原型设计流程。它们让开发者能够在几天而非几个月内迭代业务逻辑，并成为将合成标签用于训练小型专用模型的强大工具。
- 尽管 LLMs 功能高度通用，但把它们用于每一项分类任务通常既低效又昂贵。当前共识更倾向于：在开发与启动阶段使用 LLMs，随后将其知识"提炼"到更小、更快、更便宜的模型中，例如 decision models 或微调后的 transformers，以满足长期生产需求。
- "Bitter Lesson"告诉我们，通用且计算密集型的方法往往胜过专用方案，但这并不否定在软件工程中部署专用单元的现实必要性。单体式模型在延迟、资源消耗以及在通用架构里维护专用逻辑方面都面临挑战。
- 数据安全和知识产权方面的担忧常常会阻碍跨客户训练，这在过去曾迫使开发者从头开始构建模型。 LLMs 则通过提供"world knowledge"来绕开这一点，使得较小的模型也能在显著更少的专有数据下达到较高性能。
- 使用 LLMs 的开发体验很像 Electron apps：它们往往占用大量资源，架构也不够理想，但它们大幅降低了门槛，让不具备深厚 machine learning 专业背景的通用型开发者也能构建复杂功能。
- machine learning 的"研究"端与"生产"现实之间存在明显差异。虽然 LLMs 非常适合原型验证，但它们的不可预测性和高成本在规模化落地时往往会成为负担。一旦问题领域被充分理解，就需要切换到更稳定、确定性更强的系统。
- 技术社区已经识别出 LLMs 的关键可靠性问题，例如对 JSON key 排序的敏感性。这些发现也解释了为什么在没有严格验证或下游优化的情况下，把 LLMs 直接依赖在核心基础设施上，会被视为对高风险业务运营而言的高风险做法。
- 大规模的 ticket classification 系统——例如大型科技公司在使用的那些——每年处理数千万条目。在这种规模下，哪怕只是降低分类延迟或略微降低每次推理成本，都可能带来每年数百万美元的节省，因此也证明了投入定制化模型的必要性。
- 从"购买与购买"（消费专有 LLM API）的模式，转回"制造"定制化、轻量级模型，被视为重新夺回技术控制权、并减少 vendor lock-in 的必要步骤。
整体讨论反映了 AI 落地向成熟化推进的更大趋势：围绕 LLMs 的早期热潮正在受到成本、规模和可维护性等现实约束的限制。普遍观点认为，LLMs 虽然在启动和快速实验阶段价值无可替代，但在狭窄、高频的分类任务中往往属于"大材小用"。新兴的最佳实践指向一种混合工作流：先用 large generative models 来定义流程并完成标记，再把这种行为提炼为更小、更专用、更高效的架构。这样的演进表明，行业中 machine learning 的未来不取决于少数几个巨型模型，而取决于一个异构生态系统：在这里，针对特定任务选择最合适的工具——从简单的 SVM 到面向领域的 transformer 。 • The industry has seen a cyclical shift where initial efforts to build efficient, specialized machine learning classifiers were sidelined by the emergence of powerful LLMs, only to see a return to lightweight classifiers as organizations seek cost-effective, high-throughput solutions for production environments.
• LLMs have revolutionized prototyping by removing the once-prohibitive barrier of needing massive, hand-curated datasets before a model can perform useful work. They allow developers to iterate on business logic in days rather than months, serving as a powerful tool to generate synthetic labels for training smaller, specialized models.
• While LLMs are highly versatile, using them for every classification task is often inefficient and expensive. The current consensus favors using LLMs during the development and bootstrapping phases, followed by "distilling" the knowledge into smaller, faster, and cheaper models—such as decision models or fine-tuned transformers—for long-term production use.
• The "Bitter Lesson" of AI suggests that general, compute-heavy approaches often outperform specialized ones, yet this does not negate the practical necessity of deploying specialized units in software engineering. Monolithic models face challenges in terms of latency, resource consumption, and the difficulty of maintaining specialized logic within a generalized architecture.
• Data security and intellectual property concerns often prevent cross-customer training, historically forcing developers to build models from scratch. LLMs circumvent this by providing "world knowledge" that allows smaller models to achieve high performance with significantly less proprietary data.
• The development experience with LLMs is analogous to Electron apps: they consume massive resources and offer a less optimal architecture, but they significantly lower the barrier to entry, allowing generalist developers to build complex features without requiring deep machine learning expertise.
• There is a clear distinction between the "research" side of machine learning and the "production" reality. While LLMs are excellent for prototyping, their unpredictable nature and high costs often make them a liability at scale, necessitating a transition to more stable, deterministic systems once the problem space is well understood.
• The technical community has identified significant reliability issues with LLMs, such as the sensitivity of outputs to JSON key ordering. Such findings highlight why relying on LLMs for core infrastructure without rigorous validation or downstream refinement is considered risky for high-stakes business operations.
• Large-scale ticket classification systems, such as those used by major tech corporations, handle tens of millions of entries annually. At this scale, even minor optimizations in classification latency or cost per inference result in millions of dollars of annual savings, justifying the investment in custom-tailored models.
• Moving from a "buy-vs-buy" model of consuming proprietary LLM APIs back toward "making" custom, lightweight models is seen as a necessary step for regaining technical control and reducing vendor lock-in.
The discussion reflects a broader shift toward maturity in AI implementation, where the initial hype surrounding LLMs is being tempered by the practical constraints of cost, scale, and maintainability. Participants generally agree that while LLMs provide unmatched value in bootstrapping and rapid experimentation, they are often overkill for narrow, high-frequency classification tasks. The emerging best practice involves a hybrid workflow: utilizing large generative models to define and label processes, then distilling that behavior into smaller, specialized, and more efficient architectures. This evolution suggests that the future of machine learning in industry is not solely dependent on a few giant models, but rather a heterogeneous ecosystem where the right tool—from a simple SVM to a domain-specific transformer—is chosen for the specific demands of the task.