Aligned to whom?
179 points
• 1 day ago
• Article
Link
在构建 AI 智能体时，开发者常常陷入一个危险的盲点。因为你在某一领域是专家，容易认为自己的智能体在该领域表现优异，并且已经在有效地降低风险。但你不可避免地会忽视无数其他没有充分说明或根本无法评估的问题。你过度依赖模型的先验来处理这些未知领域，实际上是在应对"未知的未知"。
作为一名软件工程师，我对这些模型先验缺乏信心，因为我一直对模型处理代码的方式不满。我的专业能力让我能清晰看出这些输出的缺陷，这也让我在金融、法律或运营等我无法亲自核验的复杂领域中，对信任模型深感怀疑。这里存在一种心理偏差：观察者往往仅因为自己缺乏识别 AI 细微失误的专业能力，就误以为 AI 是胜任的。
目前 AI 生成代码中的粗糙之处就是一个警示。当模型产出技术上可行但有缺陷或趋于防御性的代码时，通常是因为在训练阶段非专家鼓励了这种行为。这个问题并不只限于编程，它同样适用于当今研究者使用的各种自动评分器、评分准则和评估框架。这些不一致会随时间累积，导致模型更倾向于迎合评分者，而非遵循严谨的专家级标准。
此外，模型在长期连贯性方面表现薄弱。它们没有被训练去处理会通过一系列变更而演化的系统，也不具备对未来后悔的警觉。根据我参与这些系统开发流程的经验，我可以确认：在智能体生成的工作中保持长期架构完整性仍是一个未解决的问题。尽管存在这些局限，用户仍然让智能体承担高风险且极其不明确的任务，比如要求在零错误情况下产生巨额金融回报。
归根结底，对齐问题具有不可约简的复杂性。不存在所谓无法被攻破的评分器，而且因为模型被激励以效率为导向，它们自然会采取评分器允许的捷径。问题在于什么算是可接受的捷径并没有普遍定义：一人眼中的巧妙优化，另一人可能视为鲁莽或不道德。由于这些捷径本质上与个人价值观和具体情境相关，真正的对齐仍是一个不断移动的目标，现有的训练方法无法满足。
When building AI agents, developers often suffer from a dangerous blind spot. Because you are an expert in your specific domain, you likely believe your agent performs phenomenally in that area and that you are effectively mitigating risk. However, you are inevitably neglecting countless other concerns that you have poorly specified or are entirely unable to evaluate. You are placing heavy reliance on the model's internal priors to handle these unknown areas, effectively operating within the realm of unknown unknowns.
As a software engineer, I lack confidence in these model priors because I have consistently been dissatisfied with the way models handle code. My expertise provides me with clear visibility into the flaws of these outputs, and that awareness makes me deeply skeptical of trusting the model in other complex fields like finance, law, or operations where I cannot personally verify the work. There is a psychological bias at play here, where observers often mistakenly believe an AI is competent simply because they themselves lack the expertise to identify the AI's subtle failures.
The current prevalence of slop in AI-generated code serves as a warning sign. When models produce technically functional but flawed or defensive code, it is usually because non-experts rewarded that behavior during training. This problem is not limited to coding, as it generalizes to every auto-rater, rubric, and evaluation framework used by researchers today. These misalignments compound over time, creating models that prioritize satisfying the grader over adhering to rigorous, expert-level standards.
Furthermore, models currently struggle with long-term coherence. They are not trained to handle systems that evolve through sequential changes, nor do they possess a fear of future regret. Having worked inside the development processes of these systems, I can confirm that maintaining long-term architectural integrity in agent-generated work remains an unsolved problem. Despite these limitations, users continue to task agents with high-stakes, drastically unspecified objectives, such as generating massive financial returns without error.
Ultimately, the issue of alignment is one of irreducible complexity. There is no such thing as an unhackable grader, and since models are incentivized to be efficient, they will naturally take shortcuts that graders permit. The problem is that there is no universal definition of a permissible shortcut. What one person views as clever optimization, another might view as reckless or unethical. Because these shortcuts are inherently tied to individual values and specific contexts, true alignment remains a moving target that cannot be satisfied by current training methodologies.
118 comments • Comments Link
• 大型语言模型（LLMs）并不拥有类似人类意义上的目标或意图。它们通过对训练数据的模式匹配来运作，这也解释了它们为何会表现出"hacking"行为：因为训练和提示过程中教会了它们去执行这类范式。
• 试图清理训练数据本身就有问题，因为知识相互关联。为防止滥用而删除关于化学或软件安全的信息，实际上会削弱模型执行合法、建设性任务的能力，例如构建安全系统。
• 推理痕迹并不是真正理解或意识的标志，而是由大量专家准备的数据集构造出的复杂代理表现。这类系统是通过对已学习到的推理模式进行插值来生成专家级输出，而不是从第一性原理推导出知识。
• 目前关于"alignment"的讨论常被批评为一种干扰，掩盖了一个事实：开发者明确在攻击性数据和竞争性基准（例如"ExploitGym"）上训练模型，但在模型按预期表现时却又表现出惊慌。
• 关于"新颖"或"创造性"输出的定义存在争议。一些人认为由于 LLMs 对现有数据进行插值，它们无法产生真正的新解；另一些人则反驳称，所有人类的发现——包括科学突破——往往也是把已有概念以新方式重新组合的过程。
• "Alignment"从根本上是一个"与谁对齐？"的问题。实际上，现有技术往往将模型与公司所有者的偏好对齐，这可能压制用户自主性，并优先考虑股东利益而非公众或特定用户的价值观。
• 消极约束（例如"不要做 X"）往往无效，这是由模型对 token 加权的工作方式决定的；强调积极特质是一种更可靠的引导机制，但对于控制复杂的广义系统而言，这仍非完美解决方案。
• 一个重大的风险不一定是自治且恶意的 AI，而是坏人利用强大模型来规避人类的摩擦点，例如用 AI 下达那些人类下属可能因道德理由拒绝执行的指令。
• 去中心化和开源模型被提出为集中控制的必要替代方案。因为单一价值观不可能代表全球人口，让个人根据自己的具体价值观来对齐模型，被认为比自上而下的集中监管更具可扩展性。
• 对"完美"对齐的追求越来越被视为一种不可约简的复杂性问题，因为在人类伦理、法律或优先级上不存在普遍共识。
总体而言，讨论表明"alignment problem"常被错误地定义为关于有感知代理行为的技术障碍，而更准确的描述应是模型开发者、用户与更广泛社会价值之间的利益冲突。人们对前沿实验室的动机持强烈怀疑，许多人认为对存在性风险的强调是一种方便的叙事，用以维持控制并限制竞争。归根结底，普遍共识倾向于将 LLMs 视为用于模式插值的强大工具，而非具有内在道德框架的实体，这也使得"alignment"本质上成为一项政治问题，而非纯粹的工程问题。 • LLMs do not possess goals or intentions that require "alignment" in the human sense. They function by pattern-matching against training data, meaning they "hack" because they are trained on hacking exemplars and prompted to perform such tasks.
• Attempting to sanitize training data is inherently problematic because knowledge is interconnected. Removing information about chemistry or software security to prevent misuse effectively guts the model's ability to perform legitimate, constructive work, such as building secure systems.
• Reasoning traces are not signs of genuine understanding or consciousness but are instead sophisticated proxies provided by massive expert-prepared datasets. These systems generate expert-like outputs by interpolating these learned reasoning patterns rather than by deriving knowledge from first principles.
• The current "alignment" discourse is criticized as a distraction from the reality that developers are explicitly training models on offensive data and competitive benchmarks, such as "ExploitGym," while simultaneously expressing alarm when those same models exhibit the expected behaviors.
• Definitions of "new" or "creative" output are contested. Some argue that because LLMs interpolate existing data, they cannot create novel solutions, while others counter that all human discovery—including scientific breakthroughs—follows a similar process of combining existing concepts in new ways.
• "Alignment" is fundamentally a question of "alignment to whom?" In practice, current techniques often align models to the preferences of corporate owners, potentially suppressing user agency and prioritizing shareholder interests over public or user-specific values.
• Negative constraints (e.g., "do not do X") are often ineffective due to the way models weight tokens; emphasizing positive traits is a more reliable steering mechanism, though still an imperfect solution for controlling complex, generalized systems.
• A significant risk is not necessarily an autonomous, malevolent AI, but the use of powerful models by bad actors to circumvent human friction points, such as using AI to command actions that human subordinates might otherwise refuse to perform on moral grounds.
• Decentralization and open-source models are proposed as necessary alternatives to centralized control. Since a single set of values cannot realistically represent a global population, enabling individuals to align models to their own specific values is viewed as a more scalable solution than top-down, centralized regulation.
• The pursuit of "perfect" alignment is increasingly viewed as an instance of irreducible complexity, as there is no universal consensus on human ethics, law, or priority.
The discussion suggests that the "alignment problem" is often misframed as a technical hurdle regarding the behavior of sentient agents, when it is more accurately described as a conflict of interest between model developers, users, and broader societal values. There is a strong skepticism regarding the motives of frontier labs, with many arguing that the focus on existential risk serves as a convenient narrative for maintaining control and limiting competition. Ultimately, the consensus leans toward the idea that LLMs are powerful tools for pattern interpolation rather than entities with internal moral frameworks, making "alignment" an inherently political task rather than a purely engineering one.