Measuring the sloppiness of code
288 points
• 3 days ago
• Article
Link
尽管 Large Language Models 在生成语法正确的代码方面已非常娴熟，但它们的普及也带来了一个隐性的代码质量危机。即便代码能通过自动化测试，往往仍存在过度抽象、冗余重复和糟糕的架构选择。此类问题通常被称为 slop，会导致代码行数失控式增长。随着项目每月新增数百万行，人类开发者已难以保持监督，而与常见观点相反，AI agents 目前也无法自我修正这种技术债务的累积。
目前业界对这一问题的评估过于依赖主观"感觉型"指标，难以产出有价值的数据。用 AI 来审查代码质量并不可靠：模型常常给出不一致的评分，或受命名等表面因素影响而改变偏好。尽管人工评估仍是保证可读性的金标准，但它缺乏现代训练流程和大规模基准所需的可扩展性。
为应对这一难题，研究者开始采用定量指标来区分 legacy codebases 与由 AI 生成的 slop 。两个较有前途的度量是 verbosity（通过启发式方法检测重复或不必要的冗长片段）和 erosion（衡量系统复杂度在多大程度上集中在少数过大、密集的函数中）。这些指标揭示了一个残酷事实：AI-generated code 在冗长度和侵蚀程度上约为 human-authored software 的两倍。
Agents 无法应对这种复杂性的现象在 SlopCodeBench 等基准中被凸显出来：这些基准通过在检查点间擦除模型上下文来模拟真实的迭代开发过程。在糟糕决策随时间累积的情形下，最先进的模型往往无法在多轮迭代中维持一个可用的 codebase 。对于那些在缺乏充分监督下大量引入 machine-generated code 的团队来说，这一失败是严重的警示。
归根结底，衡量代码质量仍离不开人类的直觉与审美。定量指标虽可帮助识别问题规模，但仅是迈向软件可维护性的一步。今后对 code churn 、 function cohesion 和 system coupledness 等指标的研究，将对完善这些工具至关重要。随着领域的发展，关注点必须从单纯生成代码，转向营造一个把结构完整性与功能性同等重视的开发环境。
While Large Language Models have become remarkably adept at generating formally correct code, their rise has ushered in a hidden crisis of code quality. Even when code passes automated tests, it often suffers from excessive abstraction, redundant duplication, and poor architectural decisions. This phenomenon, often termed slop, leads to an uncontrolled explosion of lines of code. As projects grow by millions of lines per month, human developers lose the ability to maintain oversight, and contrary to popular belief, AI agents are currently incapable of self-correcting this accumulation of technical debt.
The industry's current approach to evaluating this problem relies heavily on subjective, vibes-based metrics that fail to provide meaningful data. Using AI as a judge to grade code quality is largely ineffective, as models frequently produce inconsistent results or change their preferences based on superficial factors like naming conventions. Conversely, relying on human evaluation is the gold standard for maintaining readability, but it lacks the scalability required for modern training workflows and large-scale benchmarking.
To address this, researchers are turning to quantitative metrics to distinguish between legacy codebases and AI-generated slop. Two particularly promising measures are verbosity, which tracks duplicated or unnecessarily verbose segments using heuristics, and erosion, which calculates how much of a system's complexity is concentrated within a few oversized, dense functions. These metrics reveal a stark reality, as AI-generated code consistently proves to be roughly twice as verbose and eroded as human-authored software.
The inability of agents to manage this complexity is highlighted by benchmarks like SlopCodeBench, which simulate real-world, iterative development by erasing model context between checkpoints. Under these conditions, where bad decisions accumulate over time, state-of-the-art models often fail to maintain a functional codebase across multiple iterations. This failure serves as a critical warning for teams currently integrating vast amounts of machine-generated code into their systems without adequate oversight.
Ultimately, measuring code quality remains deeply tied to human intuition and taste. While quantitative metrics offer a way to identify the scale of the problem, they are only the beginning of a larger effort to ensure software remains manageable. Future explorations into code churn, function cohesion, and system coupledness will be essential to refining these tools. As the field evolves, the focus must shift from simply generating code to fostering an environment where structural integrity is treated with the same importance as technical functionality.
236 comments • Comments Link
• 关于编程已经"solved"的说法受到广泛质疑：目前的 AI 模型在高层架构决策、长期可维护性以及将复杂功能集成到大型既有系统方面仍然力不从心。
• 虽然 AI 能有效完成一次性任务和简单脚本，但在缺乏监督时常会产出冗余、低质量的代码，进而增加技术债务并使对最终软件进行逻辑推理变得更困难。
• 行业内缺乏衡量代码质量的稳健客观指标；单靠代码行数（LOC）等简单指标会触发 Goodhart's Law，导致模型倾向于优化指标而非真正的质量。
• 在软件开发中有效使用 AI 需要积极的人为监督，人在流程中应扮演导演或架构师的角色，而不是单纯的打字员，这通常要求高度自律与辅助工具来维护代码完整性。
• 编程在团队沟通中是一个有损通道，随着 AI 接管越来越多的实现工作，维持人类团队对系统的连贯认知变得愈发关键且艰难。
• 企业级软件开发受制于遗留系统、业务需求和高可靠性要求等复杂约束，目前的 agents 通常无法自主管理这些需求。
• AI 编码代理在"在正确的地方进行修改"方面表现欠佳，常常违背既定架构模式，因为它们缺乏对全局设计和原始系统意图的深入理解。
• 对 AI 的看法存在显著分歧：一方把它视为变革性的生产力工具，认为通过提高可能性的下限从而从根本上"solved"了编程；另一方则强调，软件工程的核心挑战——可靠性、安全性与长期维护——尚未解决，仍需深厚的人类专业知识。
• 一些开发者认为人类编写的企业级代码历来质量堪忧，暗示尽管 AI 目前有缺陷，最终可能提升平均水准。
• 当前的 AI 热潮导致代码量呈指数级增长，但维护和调试 AI 生成系统的长期成本正在显现，可能掩盖短期开发速度带来的收益。
关于编程是否已被"solved"的争论，本质上源于对"软件开发"定义的根本分歧。一种观点把编程看作产出功能性结果的行为，从这个角度来看，AI 能生成可运行（尽管有时臃肿）的代码是一场决定性胜利。另一种观点则认为编程只是软件工程的一部分，软件工程还包括为可维护性而设计、保证可靠性以及应对复杂的组织与业务需求等更艰巨的任务。 AI 对熟练开发者是强大的倍增器，但业界普遍认为它无法替代人类在架构与问题解决上所需的全面理解，尤其是在关键的生产环境中。这场讨论反映了 AI 驱动开发带来的即时速度与传统以人为本的精准性、设计连贯性及长期系统健康价值之间的紧张关系。 • The premise that coding is "solved" is widely contested, as current AI models struggle with high-level architectural decisions, long-term maintainability, and the complex integration of features into large, existing systems.
• While AI effectively handles one-shot tasks and simple scripts, it often produces "slop" or verbose, low-quality code when left unsupervised, which can lead to increased technical debt and difficulty in reasoning about the resulting software.
• The industry currently lacks robust, objective metrics for code quality; relying on simple indicators like lines of code (LOC) often triggers Goodhart's Law, where models optimize for the metric rather than genuine quality.
• Effective AI usage in software development requires active human oversight, where the human acts as a director or architect rather than a mere typist, often necessitating significant discipline and secondary tooling to maintain code integrity.
• Coding is a lossy channel for team communication; as AI takes over more of the implementation, maintaining a coherent mental model of the system becomes a critical, yet increasingly difficult, challenge for human teams.
• Enterprise-level software development involves complex constraints like legacy systems, business requirements, and high-stakes reliability needs, which current agents are generally not equipped to manage autonomously.
• AI coding agents struggle with "making changes in the right places," often working against established architectural patterns because they lack a full understanding of the global design and the underlying intentions of the original system.
• There is a notable divide between those who view AI as a transformative productivity tool that has fundamentally "solved" coding by raising the floor of what is possible, and those who emphasize that the core challenges of software engineering—reliability, security, and long-term maintenance—remain unsolved and require deep human expertise.
• Some developers argue that human-written enterprise code has historically been of poor quality, suggesting that AI might eventually improve the average standard despite its current shortcomings.
• The current AI boom has led to an exponential increase in the volume of code, but the long-term cost of maintaining and debugging AI-generated systems is an emerging concern that may overshadow the short-term gains in development speed.
The debate over whether coding is "solved" hinges on a fundamental disagreement regarding the definition of software development. One perspective views coding as the act of producing functional output, where AI's ability to generate working, albeit sometimes bloated, code represents a decisive victory. Conversely, others argue that coding is merely a subset of software engineering, which encompasses the much harder tasks of designing for maintainability, ensuring reliability, and navigating complex organizational and business requirements. While AI acts as a powerful multiplier for skilled developers, the consensus among practitioners is that it lacks the holistic understanding necessary to replace the human role in architecture and problem-solving, particularly in critical production environments. The discussion reflects a tension between the immediate velocity of AI-driven development and the traditional, human-centered values of precision, design coherence, and long-term system health.