Real-SWE: Benchmarking AI models on private, real-world, enterprise codebases
271 points
• 1 day ago
• Article
Link
Real-SWE 是一个新的基准，用来评估前沿 AI 模型在私有、真实企业代码库（而非公共数据集）上的表现。该基准使用来自真实公司的授权代码，捕捉到生产工程的真实复杂性——包括专有系统、关键业务流程和公司特有的编码约定。由于 99% 的企业代码片段对公共互联网不可见，这些任务构成了对 AI 代理的真正分布外测试，要求它们完成具有实际业务后果的工作，例如税务计算和客户迁移。
基准结果表明，尽管模型能应对部分编码挑战，但距离达到专业软件工程所要求的一致性还有很大差距。在当前测试中，即便表现最好的模型 Fable 5.1，解决率也仅为 38.8%，许多其他模型远低于此水平。值得注意的是，10 个任务中有 6 个的解决率低于 15%，凸显出现阶段的 AI 代理在处理需要理解既有业务逻辑的复杂跨职能代码库时仍存在显著困难。
对失败模式的分析显示，缺失需求是最常见的问题，其次是集成错误和未经验证的假设。模型往往不是因为不够尝试而失败，而是在未检查工作区的情况下对系统做出错误猜测，或无法将新代码正确接入现有的复杂环境。此外，数据还表明，每个任务投入更高的成本并不保证成功：部署成本与模型达到正确解决方案的能力之间并无直接相关性，这表明瓶颈更可能出在架构设计而非计算预算上。
为了让评估贴近工程师的真实工作方式，基准在隔离的沙盒中使用原生 harnesses 测试模型与 harness 的组合。任务设定略显不完备，以反映现实需求，且经常需要对多个文件进行修改来满足业务要求。通过将模型放在实际生产环境中测试，Real-SWE 揭示了当前 AI 能力与真实企业软件团队所需严格标准之间的明显差距。
Real-SWE is a new benchmark designed to evaluate frontier AI models on private, real-world, enterprise codebases rather than public datasets. By utilizing code licensed from actual companies, the benchmark captures the genuine complexity of production engineering, including proprietary systems, business-critical workflows, and company-specific coding conventions. Because 99% of enterprise code tokens remain hidden from the public internet, these tasks serve as a true out-of-distribution test for AI agents, pushing them to perform work with tangible business consequences like tax calculations and customer migrations.
The benchmark demonstrates that while models can handle certain coding challenges, they are far from achieving the consistency required for professional software engineering. In current testing, even the top-performing model, Fable 5.1, achieved a resolution rate of only 38.8%, with many other models falling well below that mark. Notably, 6 out of 10 tasks saw resolution rates below 15%, highlighting that contemporary AI agents still struggle significantly when tasked with navigating complex, cross-functional codebases that require understanding existing business logic.
Analysis of failure patterns reveals that missing requirements is the most frequent issue across most models, followed by integration errors and unverified assumptions. Rather than failing due to a lack of effort, models often make incorrect guesses about the system without checking the workspace or struggle to properly wire new code into existing, multifaceted environments. Furthermore, the data indicates that higher spending per task does not guarantee success. There is no direct correlation between the cost of a rollout and the model's ability to reach a correct resolution, suggesting that architectural limitations, rather than compute budget, are the primary bottleneck.
To ensure the evaluation reflects how engineers actually work, the benchmark uses native harnesses to test model-and-harness combinations in isolated sandboxes. The tasks are slightly underspecified, mirroring real-world requirements, and frequently demand changes across multiple files to satisfy business needs. By comparing models against actual production environments, Real-SWE exposes a significant gap between current AI capabilities and the rigorous standards required by real-world enterprise software teams.
154 comments • Comments Link
• 用户质疑当前 coding benchmarks 的有效性，指出其结果常常与实际性能、速度和可用性不符。
• 一个一致的主题是"harness"（环境与工具集成）发挥着关键作用，其对模型成功的影响往往超过 LLM 本身的能力。
• 关于"decisive"模型（如 Astra，偏向速度）与"cautious"模型（如 Fable，提供更强的架构监督但可能导致决策疲劳）之间的权衡存在重大争论。
• 许多参与者报告称，模型性能对 codebase 的具体性质高度敏感：与复杂或特定的遗留系统相比，CRUD 应用更容易实现自动化。
• 对 benchmarks 中使用的专有代码来源存在广泛担忧，许多人怀疑模型是在泄露的内部代码库上训练的，从而使"私有"代码实际上变得公开。
• 用户满意度的差异（即有人认为模型能胜任，而另一些人认为不能）通常归因于提示工程的规范性、项目结构以及任务定义是否清晰。
• 一些用户认为当前处于类似"centaur chess"的阶段：人类在引导、审查和限定模型行为方面的能力，是项目成败的主要决定因素。
• 由于存在"benchmaxxing"以及源材料透明度不足，benchmarks 的可靠性受到质疑，导致很多人更倾向于依赖个人测试而非综合评分。
• 频繁出现的模型故障（如幻觉式的编码模式或循环行为）被视为主要痛点，需要持续的人为干预和保护措施。
• 一些参与者建议，现代 benchmarks 应转向评估一致性和对 harness 的依赖表现，而不是仅以静态任务的一次性成功率为准。
此次讨论反映出标准化性能基准与将 AI 代理用于专业软件工程时那种细致入微、常常令人沮丧的现实之间日益扩大的鸿沟。尽管部分开发者通过严格限定范围并使用定制化的 harness 实现了较高的成功率，但也有人发现这些工具不稳定、过度工程化，或容易陷入无谓的"rabbit holes"。大家普遍达成共识：即便模型在不断改进，人类操作者定义清晰任务并审查中间输出的能力仍然是瓶颈。总体上，社区倾向于把这些工具视为需要大量引导的 copilots，而非能取代人类判断的自主替代品。 • Users express skepticism regarding the validity of current coding benchmarks, noting that results often contradict real-world performance, speed, and usability.
• A consistent theme is the critical role of the "harness"—the environment and tool integration—which often dictates model success more than the underlying intelligence of the LLM itself.
• Significant debate exists over the trade-off between "decisive" models like Astra, which prioritize speed, and "cautious" models like Fable, which provide better architectural oversight but cause decision fatigue.
• Many participants report that model performance is highly sensitive to the specific nature of the codebase, with CRUD applications proving significantly easier to automate than complex or idiosyncratic legacy systems.
• There is widespread concern regarding the provenance of proprietary code used in benchmarks, with many suspecting that models are trained on leaked internal codebases, rendering "private" code effectively public.
• Discrepancies in user satisfaction—where one person finds a model capable and another finds it incapable—are often attributed to differences in prompting discipline, project structure, and the presence of clear task definitions.
• Some users argue that the current era resembles "centaur chess," where the human's ability to guide, review, and manage the model's scope is the primary determinant of a successful project outcome.
• The reliability of benchmarks is questioned due to "benchmaxxing" and a lack of transparency regarding the source materials, leading some to prioritize personal testing over aggregate performance scores.
• Frequent model failures, such as hallucinated coding patterns or looping behaviors, are identified as major pain points that necessitate constant human intervention and guardrails.
• Several participants suggest that modern benchmarks should move toward evaluating consistency and harness-dependent performance rather than relying on single-pass success rates on static tasks.
The conversation reflects a growing divide between standardized performance benchmarks and the nuanced, often frustrating reality of using AI agents for professional software engineering. While some developers achieve high success rates by strictly managing scope and using tailored harnesses, others find the tools inconsistent, over-engineered, or prone to aimless "rabbit holes." There is a strong consensus that the human operator's ability to define clear tasks and review intermediate outputs remains the bottleneck, even as models improve. Ultimately, the community leans toward viewing these tools as copilots that require significant guidance, rather than autonomous replacements for human judgment.