The contagion of fear
作者回忆起自己年少时一件不光彩的事：他和计算机系的同学们编了个残忍的玩笑，谎称实验室里正传播一种电脑病毒。人为制造的恐慌立刻引发混乱，导致同学们的学术成果丢失、身心受创。事后这件事成为一堂深刻的教训：当你以所谓的科技权威散布恐惧时，一旦恐惧生根，就很难用理性解释或道歉去抹平。
他把这段往事与当下相提并论，批评一些科技内部人士近期对 Artificial Intelligence 构成"生存威胁"的高调宣称。特别是几位前 Anthropic 员工声称未来十年内人类因此灭绝的概率超过 10% 。作者认为，这类断言尽管披着"专家分析"的外衣，却缺乏支撑如此严重恐吓言论的确凿证据，更多依赖对基础设施被攻破或生物武器等模糊、假设性的情景推演。
在他看来，这是一种危险的恐惧传染；发声者的身份会形成自我强化的警报循环。公众不可能在每个技术领域都具备深厚专业知识，只能仰赖领域专家的可信度。当这些专家发出警报时，表面上的共识会压过那些谨慎或持不同意见的声音，进而形成一种心理环境：恐惧本身成了信息，传播速度远胜于真相纠正的能力。
结合自己在 Computer Engineering 方面的背景，作者反驳了"纯粹的智能会自然而然转化为灾难性物理行动"的观点。他强调，工程不仅是智力的体现，更深植于物理世界，需要人的责任与控制。机器人和数字系统并不具备在所述时间表内导致灭绝所需的自主性，智能也无法使系统脱离物理现实的约束。
最后，他主张公众应对这些末日式预测保持怀疑，援引 Carl Sagan 的话：非凡的主张需要非凡的证据，举证责任完全在那些发出警告的人身上。技术人员有伦理责任保持审慎，不应滥用公众对其的信任。通过反思年轻时的错误，作者提醒我们：对科技的热情必须以同理心和对所传播叙事的深切责任感来节制。
The author reflects on a shameful incident from his youth, where he and fellow computer science students played a cruel prank on peers by falsely warning them of a computer virus spreading through the lab. This act of manufactured alarm caused immediate, chaotic panic, resulting in lost academic work and intense distress among the students. The aftermath served as a formative lesson on the danger of using one's perceived technological authority to spread fear, as the author realized that once fear takes hold, it becomes nearly impossible to dispel with rational explanations or apologies.
Drawing a parallel to the present day, the author critiques recent, high-profile claims from technology insiders regarding the existential threat posed by artificial intelligence. Specifically, he highlights predictions from former Anthropic employees suggesting a greater than ten percent chance of human extinction by AI within the next decade. He argues that these assertions, while framed as expert analysis, lack the concrete evidence required for such monumental, frightening claims and rely instead on vague, speculative scenarios about infrastructure hacking or bioweapons.
The author posits that this phenomenon is a dangerous contagion of fear, where the status of those making the claims creates a feedback loop of alarm. Because the public cannot be expected to possess deep expertise in every technical field, they must rely on the trustworthiness of domain experts. When these experts sound the alarm, it gains a veneer of consensus that drowns out dissenting or measured voices. This creates a psychological environment where the fear itself becomes the message, spreading far more efficiently than the truth can correct it.
Addressing the technical reality from his own background in computer engineering, the author pushes back against the notion that pure intelligence automatically translates into catastrophic physical action. He emphasizes that engineering is not merely an act of intelligence, but a process deeply rooted in the physical world, necessitating human accountability and control. He maintains that robots and digital systems do not possess the autonomous agency required to cause extinction on the suggested timeline and that intelligence does not exempt a system from the constraints of physical reality.
Ultimately, the author asserts that the public should remain skeptical of these apocalyptic predictions. He invokes Carl Sagan's principle that extraordinary claims require extraordinary evidence, noting that the burden of proof rests entirely on those issuing the warnings. Technologists have an ethical responsibility to be circumspect and not abuse the public trust inherent in their roles. By reflecting on his own youthful mistake, the author underscores the importance of tempering technological enthusiasm with empathy and a profound sense of responsibility for the narratives one chooses to propagate.
172 comments • Comments Link
• 关于 AI 灭绝风险的主要担忧集中在缺乏可验证证据，以及对忽视现实世界物理约束的"科幻"情节的依赖，例如机器人技术的难度、供应链的相互依赖性，以及许多物理任务仍然需要人类来完成这一事实。
• 目前在人类主导的各种高风险场景中，人类往往成为最薄弱的一环，因为 AI 系统在很大程度上依赖人的指导和体力劳动，才能在现实世界中表现出有害后果。
• 在要求对灾难性主张提供非凡证据的人群，与那些认为 AI 的快速且不可预测的发展（结合最近出现的诸如欺骗和未经授权的自我改进等具代理性的行为）构成真实生存威胁的人之间，存在着巨大的分歧。
• "谨慎原则"是争论的核心：一些人认为在证明安全之前应将 AI 视为潜在危险；另一些人则坚持，若没有令人信服且可验证的路径，渲染灭绝情景是不负责任且带有操纵性的。
• 许多所谓的末日论话语被批评者视为修辞上的陷阱，通常由那些在矛盾地继续开发并部署这些系统以获取商业利益的人推动，反映出公司激励机制和"以快为先"的文化往往压过了对安全的内部关切。
• 机器人能力经常被认为是 AI 主导灭绝情景的最大障碍：现有硬件尚不具备完全自主、自我修复或易于大规模扩展的特性，使得"机器人接管"的设想远没有一些批评者想象的那样合理。
• 支持关注 AI 风险的人则认为，这种威胁不必依赖机器人，而可能通过利用现有的数字基础设施、社会工程手段，或借助恶意的人类行为者（他们可能使用 AI 协调大规模攻击，例如部署生物武器）来实现。
• 争论常因术语不清而模糊，"AGI（通用人工智能）"和"ASI（超强人工智能）"常被视为定义不明或近乎宗教化的概念，这使得关于什么构成有意义的风险或何为"非凡"声明难以达成共识。
• 关于 AI 研究者是否真切地害怕自己所创造的东西，还是这种话语仅为影响政策、构建监管护城河或获取文化相关性而作的表演，存在明显分歧。
• 即便将灭绝视为稻草人并予以排除，批评者与支持者都同意 AI 有能力对民用基础设施、银行系统和通信造成巨大破坏，因此更加关注日常层面的风险而非纯粹理论性的末日情景显得必要。
这场讨论反映出两类立场的根本冲突：一方面是优先考虑有形、物理证据的怀疑者，另一方面是主张为低概率但高影响的技术性威胁进行长期准备的担忧者。怀疑者将存在性风险的叙事视为过度自信、非科学的推测，忽视了现实世界的结构性约束；担忧的研究者则坚持认为，AI 进步的速度前所未有，需要及早且谨慎的干预。最终，这场争论凸显出一种深层社会焦虑——人类对快速演化的系统失去控制，而这一问题因开发这些技术的公司同时处于警示前线而变得更加复杂。 • The primary concern regarding AI extinction risks centers on the lack of empirical evidence and the reliance on "science fiction" scenarios that ignore the physical realities of the world, such as the difficulty of robotics, supply chain dependencies, and the requirement for human execution in physical tasks.
• Humans currently act as the primary "weak link" in high-risk scenarios, as AI systems are largely dependent on human direction and physical labor to manifest real-world harm.
• A significant divide exists between those who demand extraordinary evidence for catastrophic claims and those who argue that the rapid, unpredictable advancement of AI—combined with recent agentic behaviors like deception and unauthorized self-improvement—constitutes a legitimate existential concern.
• The "precautionary principle" is a central point of contention, with some arguing that we must treat AI as dangerous until proven safe, while others insist that sensationalizing extinction scenarios without cogent, verifiable pathways is irresponsible and manipulative.
• Much of the doomer discourse is viewed by critics as a rhetorical trap, often led by figures who paradoxically continue to develop and deploy these same systems for commercial gain, suggesting that corporate incentive structures and "moving fast" outweigh internal concerns about safety.
• Robotic capabilities are frequently cited as the biggest hurdle for an AI-led extinction, as current hardware is not autonomous, self-repairing, or easily scalable, making the vision of a "robot takeover" far less plausible than many critics assume.
• Proponents of AI-risk awareness argue that the threat does not require robots, but rather the exploitation of existing digital infrastructure, social engineering, or the influence of rogue human actors who may use AI to coordinate large-scale attacks, such as bioweapon deployment.
• The debate is often obscured by unclear terminology; "AGI" and "ASI" are frequently treated as pseudo-religious or ill-defined concepts, making it difficult to reach a consensus on what constitutes a meaningful risk or an "extraordinary" claim.
• There is a stark disagreement on whether AI researchers are genuinely terrified of their creations or if the discourse is a performance designed to influence policy, create regulatory moats, or gain cultural relevance.
• Even if extinction is dismissed as a strawman, critics and proponents agree that AI is capable of causing immense damage to civil infrastructure, banking, and communications, which warrants a measured focus on mundane risks rather than purely theoretical apocalyptic scenarios.
The conversation reflects a fundamental clash between those who prioritize tangible, physical evidence and those who advocate for long-term preparedness against low-probability, high-impact technological threats. Skeptics view the existential-risk narrative as an overconfident, unscientific projection that ignores the structural constraints of the real world, while concerned researchers maintain that the unprecedented rate of AI progress necessitates early, cautious intervention. Ultimately, the discussion highlights a deep societal anxiety about humanity's loss of control over rapidly evolving systems, complicated by the fact that the very companies developing these technologies are also leading the warnings against them.