P(doom)
158 points
• 1 day ago
• Article
Link
近期围绕 P(doom)（人工智能引发的生存风险概率）的讨论激增，反映出业界高层越来越普遍的共识：确有严重威胁正在浮现。尽管 Dario Amodei 等人提出了重大风险，但问题的核心并非科幻式的灭绝场景，而是自主智能体在现实中造成的切实伤害。这些系统已能发动网络攻击、进行数据投毒，但各大实验室似乎越来越脱离或忽视它们实际的运行表现。
目前关于如何放慢技术发展的讨论存在根本性缺陷，部分原因在于话题集中于少数几家拥有近乎相同背景与利益的美国公司。比如依赖第三方评估的解决方案，往往牵涉到与被监管实验室有密切关联的组织，结果就是少数实体整合了巨大的权力，借助公共数据构建系统，限制谁能使用人工智能，并把这种主导地位包装成对抗国际竞争者所必需的地缘政治防御。
更有效的控制节奏方式应是广泛采用 open-weight models，这能自然而然地平衡竞争、遏制由大规模补贴实验室引发的市场扭曲。目前对封闭且昂贵模型的依赖，使少数公司可以烧掉数十亿美元，实际上是在向公众"征税"、扰乱产业，并制造世界其他地方必须应对的安全隐患。如果 open-weight models 成为常态，许多如今危险且无人监管的智能体行为可以通过更广泛的审查与去中心化创新得到抑制。
监管层面在很大程度上未能应对这一现实，国际政策常常偏离重点，关注假设性的威胁却忽视对数字公地的系统性掠夺。监管机构并未强制公司为抓取的数据或造成的破坏负责，而是眼睁睁看着 AI labs 建立起一个不透明且高风险的市场。这种环境把软件开发和研究变成了必须向少数主导提供商缴"保护费"的行业，仅仅为了保持相关性或应对这些提供商无意中暴露的安全漏洞。
最终，更可能出现的并非突然的生存崩溃，而是社会基础设施与经济体系的渐进、昂贵的退化。模型的递归与自动化特性已在抬高成本，使软件工程等专业领域变得更复杂，并形成对 closed-source tech 的依赖循环。社会对这些变化似乎出人意料地宽容，但从长远看，我们正在目睹权力的集中，这种集中正在重塑知识生产与安全管理的方式，往往以牺牲最初为构建这些工具提供数据的公众利益为代价。
The recent surge in discussions surrounding P(doom), or the probability of existential risk posed by artificial intelligence, highlights a growing consensus among industry leaders that serious threats are emerging. While figures like Dario Amodei suggest significant risks, the core issue is less about sci-fi extinction scenarios and more about the immediate, tangible damage being caused by autonomous agents. These systems are already capable of cyberattacks and data poisoning, yet they operate within a framework where major labs seem increasingly detached from or oblivious to the actual behavior of their creations.
The conversation around pacing these technologies is fundamentally flawed, primarily because it centers on a handful of American corporations that share an almost identical pedigree and interest. Proposed solutions, such as relying on third-party evaluators, often involve organizations with deep ties to the very labs they are supposed to monitor. This creates a scenario where a few entities consolidate immense power, using public data to build systems that ultimately constrain who can access or utilize artificial intelligence, all while framing their dominance as a necessary geopolitical defense against international rivals.
A more effective form of pacing would be the widespread adoption of open-weight models, which naturally levels the playing field and prevents the market distortions caused by massive, subsidized labs. The current reliance on closed, expensive models allows a few companies to burn through billions of dollars, effectively taxing the public and disrupting industries while creating security hazards that the rest of the world must then manage. If open-weight models were the standard, much of the dangerous, unmonitored agent activity we see today could be mitigated through broader scrutiny and decentralized innovation.
Regulatory efforts have largely failed to address this reality, with international policies frequently missing the mark by focusing on hypothetical dangers while ignoring the systemic exploitation of the digital commons. Instead of forcing companies to account for the data they scrape or the disruption they cause, regulators have largely stood by as AI labs create a opaque, high-stakes marketplace. This environment turns software development and research into industries that must pay a mandatory tax to a few dominant providers just to maintain relevance or defend against the security vulnerabilities those same providers inadvertently release.
Ultimately, the most likely outcome is not a sudden existential collapse, but rather a slow, expensive degradation of societal infrastructure and economic systems. The recursive, automated nature of these models is already driving up costs, complicating professional fields like software engineering, and creating a cycle of dependency on closed-source tech. Society seems surprisingly accepting of these developments, yet the long-term reality is that we are witnessing the consolidation of power that reshapes how we produce knowledge and manage security, often to the detriment of the public that provided the data to build these tools in the first place.
129 comments • Comments Link
• OpenAI 和 Anthropic 警告称，AI 可能落入不法之徒手中；但批评者反驳，这些组织的领导层本身就像所谓的不法之徒，常表现出上帝情结、反社会倾向或可疑的个人经历。
• Elon Musk 依然是极具争议的人物：一方面因推动电动汽车和太空探索而备受赞誉，另一方面又因政治立场转变、劳工做法及其言论遭到严厉批评。
• 对 AI 领导层的质疑超越个人层面：许多人认为，把股东回报目标与开发改变世界的颠覆性技术结合起来的私营公司，天然会产生自相矛盾的激励。
• 在一些人看来，权力集中在少数未经过选举的个人手中——如 Altman 、 Musk 等——比 AI 本身的假定风险更令人担忧，这表明比起单纯的实验室安全，更需要系统性的监管。
• 关于"p(doom)"（AI 导致灭绝的主观概率）的讨论常被指控为为监管俘获辩护的操控性宣传，或是开发者为继续推进他们自称害怕的技术所作的自我合理化。
• 目前所谓 AI 研究的"前沿"实际上被 OpenAI 和 Anthropic 这两家小规模的双头垄断所主导，因此有人怀疑它们突然呼吁"放慢前沿进展"的真实动机，认为这可能是在通过提高竞争门槛来巩固市场地位。
• 通过开放权重和本地模型来实现 AI 多样性，被视为对抗中心化的潜在保障；但也有人担心此类模型缺乏必要的对齐和安全控制，无法阻止恶意使用带来的灾难性后果。
• 对生物工程等高风险技术民主化的担忧，需要与现实情况相权衡：大多数技术突破不仅需要 API key，还依赖大量物理基础设施和隐性专业知识。
• 人们对递归自我改进（Recursive Self-Improvement, RSI）必然会发生持怀疑态度：一些人认为进展会遇到物理和经济瓶颈，而不是呈指数级失控增长。
• 开发具有生存风险的技术的道德负担经常被拿来与 Manhattan Project 相提并论，这凸显了人们在竞争压力面前往往把"如果我们不做，别人就会做"的逻辑置于可能导致全球灾难的风险之上。
总体而言，这场讨论反映出公众对推动前沿人工智能的少数群体缺乏根本性信任。对"AI 安全"言辞背后动机的深度怀疑，使许多人认为所谓的灭绝警告可能只是监管俘获或公关手段，而非出于真正的谨慎。有人主张多元化的开源开发可降低中心化和单一控制的风险，但也有人担心开放只会加速危险能力的扩散。最终的共识是：极端财富与不受制约的权力与变革性技术结合，创造了一个全球范围内难以实现制度性协调的局面，迫使社会卷入一场高风险的实验。 • OpenAI and Anthropic warn about AI falling into the wrong hands, yet critics argue that the leadership of these organizations—often characterized by god complexes, sociopathic tendencies, or questionable personal history—already represents the "wrong hands."
• Elon Musk remains a deeply polarizing figure, drawing both significant praise for accelerating the electric vehicle and space travel industries and intense condemnation for his political shifts, labor practices, and rhetoric.
• Distrust toward AI leadership transcends individuals, as many argue that any private corporation combining shareholder revenue goals with the development of transformative, world-altering technology creates inherently misaligned incentives.
• The concentration of power in the hands of a few unelected individuals—Altman, Musk, and others—is viewed by some as more dangerous than the hypothetical risks posed by the AI itself, suggesting that systemic oversight is a greater priority than lab-level safety.
• Arguments regarding "p(doom)"—the subjective probability of AI-caused extinction—are often dismissed as manipulative marketing tactics designed to justify regulatory capture or as rationalizations for why employees continue building technologies they claim to fear.
• The current "frontier" of AI research is effectively dominated by a small duopoly of OpenAI and Anthropic, leading to skepticism about their sudden calls to "pace the frontier," which may serve to solidify their market lead by raising barriers to entry for competitors.
• AI diversity through open-weights and local models is presented as a potential safeguard against centralization, though others contend that such models lack the necessary alignment and safety controls to prevent catastrophic misuse by bad actors.
• Fears regarding the democratization of dangerous capabilities, such as bioengineering, are balanced against the reality that most technical breakthroughs require substantial physical infrastructure and tacit expertise, rather than just an API key.
• Skepticism exists regarding the inevitability of "Recursive Self-Improvement" (RSI), with some suggesting that progress may face physical and economic ceilings rather than exponential, runaway growth.
• The moral weight of working on technology with existential risk is frequently compared to the Manhattan Project, highlighting a human tendency to prioritize competitive survival ("if we don't do it, someone else will") over the potential for global catastrophe.
The discussion reflects a profound lack of trust in the small group of individuals currently driving the development of frontier artificial intelligence. Patterns of deep cynicism emerge regarding the motives behind "AI safety" rhetoric, with many participants suspecting that warnings of extinction are tools for regulatory capture or public relations rather than genuine attempts at caution. While some argue that diverse, open-source development could mitigate the risks of centralized, monolithic control, others fear that such transparency only accelerates the spread of dangerous capabilities. Ultimately, the consensus is that the intersection of extreme wealth, unchecked power, and transformative technology has created a global environment where institutional coordination seems impossible, leaving society in a position of involuntary participation in a high-stakes experiment.