A false smoke alarm almost drowned Jane Street's only NYC datacenter
567 points
• 1 day ago
• Article
Link
Jane Street 的技术与基础设施思路很不寻常：他们倾向于把关键硬件直接安置在曼哈顿办公室内部。这个选择既出于对交易策略安全性的担忧，也出于务实的考虑——他们希望能对自动化系统保持物理层面的掌控，必要时甚至可以手动拔掉电源线。公司文化受一位有核潜艇背景的合伙人影响，更偏好可触摸、机械式的失效保护，而非完全依赖纯自动化。于是，系统团队既是工程师，也要在技术故障发生时第一时间顶上处理。
这种对本地化基础设施的依赖，在隔壁房间发生一场施工相关事故、从而触发误报火警时遭遇了严峻挑战。火灾报警本身是标准防护措施，却引发了连锁反应，差点危及整个 datacenter 。系统在检测到烟雾后会自动关闭计算机机房用精密空调（CRAC）单元以阻止火势蔓延，于是机房等同于失去了冷却能力。喷淋系统已加压并随时可在达到热阈值时启动，但机房里热量无法被有效带走，温度开始迅速上升。
作者在接到香港办公室的紧急电话后，因住处离办公室仅需步行几分钟，立刻赶去现场救急。这已然变成一场与时间赛跑：机房内的环境温度正向临界点逼近，随时可能触发感温喷淋头。到达后，作者当机立断，手动逐一断开设施内的所有电源线。这样的直接物理处置让温度在即将造成更严重后果的前几刻得以稳定，避免了核心硬件遭受灾难性的水损。
事故之后的善后处理同样保持冷静与工程导向，重点在于修补系统层面的薄弱环节，同时不放弃公司独特的物理布局。团队没有追究责任人，而是复盘整起事件，以便完善流程并加强监控。最关键的改进，是建设了一个更可靠的监测系统。鉴于原有遥测数据不够稳定，作者开发了一个基于 OCaml 的监控工具：它直接向每台 Linux 服务器查询温度传感器，确保团队能在任何局部或整体的冷却故障把情况推到危险阈值之前就收到告警。
Jane Street's approach to technology and infrastructure was deeply unconventional, characterized by a preference for keeping critical hardware housed directly within their Manhattan office space. This decision was driven by both security concerns regarding trading strategies and a pragmatic desire to maintain physical control over automated systems, which included having the ability to manually pull power cords if needed. The firm's culture, influenced by a partner with a nuclear submarine background, favored tangible, mechanical fail-safes over purely automated ones, creating an environment where the systems team acted as both engineers and front-line responders to any technical disruption.
The reliance on this localized infrastructure was severely tested when a construction-related incident in a neighboring room triggered a false smoke alarm. While the fire alarm system was a standard precaution, it set off a chain reaction that threatened the entire datacenter. Because the system was designed to automatically shut down Computer Room Air Conditioner (CRAC) units upon detecting smoke to prevent fire spread, the room was effectively deprived of its cooling capacity. With the sprinkler system pressurized and ready to deploy if heat thresholds were met, the datacenter sat in a rapidly warming environment with no way to dissipate the heat generated by the servers.
Receiving an urgent call from the Hong Kong office, the author, who lived just a short walk away, rushed to the office to intervene. The situation had become a race against time, as the ambient temperature in the machine room began to climb toward the critical point that would trigger the heat-sensitive sprinkler heads. Upon arriving, the author scrambled to manually disconnect every power cable in the facility. This drastic physical intervention succeeded, stabilizing the temperature just moments before it could have resulted in catastrophic water damage to the firm's core hardware.
Following the event, the aftermath was managed with a level-headed, engineering-focused approach that aimed to address systemic weaknesses without abandoning the firm's unique physical layout. Rather than assigning blame, the team analyzed the incident to refine their procedures and surveillance. The most significant outcome was the development of a more reliable monitoring system. Recognizing that existing telemetry was inconsistent, the author developed an OCaml-based monitoring tool that queried the temperature sensors of every Linux server directly, ensuring that the team would be alerted to any localized or generalized cooling failure before it reached a dangerous threshold.
268 comments • Comments Link
• Hurricane Sandy 暴露了数据中心基础设施的关键漏洞，迫使员工完成类似"超人"般的任务，例如当地下室泵失效时，徒手把柴油搬上近 20 层楼梯，以维持服务器运行。
• 依靠个人英雄主义去缓解系统性故障，常被视为领导力不足和规划欠缺的表现，而不是真正的成功。
• 工作场所文化很容易走向对员工奉献精神的剥削，把"英雄主义"当作替代方案：少招人或不投资适当冗余的廉价做法。
• 定义灾难恢复（Disaster Recovery, DR）并不简单。一些组织选择通过接受低频、高强度的手动介入来优先节省成本；而另一些组织则认为，缺乏稳健的故障切换（Failover）本质上就是专业层面的失败。
• 行业内常常把紧急情况下的"英雄主义"当作典范，可能因此掩盖更深层的问题，例如未能预见风险，或缺少像 Shunt trips 这类用于紧急断电的现代化基础设施。
• 划清个人生活与职业义务的边界至关重要，因为雇主可能会拿员工的私人情况或办公地点距离作为筹码，要求其承担无偿且超常的劳动。
• 关于数据中心位置与设计的决策，通常需要在成本、 Latency（延迟）和环境风险之间做权衡。对这些决策的事后批评，必须同时考虑当时领导者所掌握的有限信息。
• 设在办公楼内的数据中心，其 Fire suppression systems 往往依赖 Sprinklers（喷淋系统）。这通常源于建筑规范或安全考量，却也带来高风险：一旦误报发生，可能就需要进行疯狂的手动断电操作，以避免水渍损坏。
• 激烈的市场竞争压力促使公司最大化员工产出，结果往往是更看重短期强度，而不是长期稳定性和员工健康。
• 对故障的"英雄式"应对固然在个人层面令人敬佩，但理想情况下，它应当推动严谨的系统性调查，确保今后再也不必采取这种超常举措。
这场讨论反映了"英雄式"工程壮举的浪漫化叙事与系统性疏忽的现实之间的拉扯。尽管许多人都认可在 Hurricane Sandy 等灾难中挺身而出的个人奉献精神，但共识很明确：这些努力往往掩盖了长期在基础冗余和灾难规划上的投入不足。对话也凸显了企业文化如何剥削个人的承诺——高压场景确实能展示个体能力，但同样也暴露出领导力层面的失误。最终大家更倾向于这样一种理念：真正的成功，是打造具备弹性的系统，使其在可预见的危机中无需"超人"般的临时介入也能运转。 • Hurricane Sandy exposed critical vulnerabilities in data center infrastructure, forcing staff to perform "superhuman" tasks like carrying diesel fuel up nearly 20 flights of stairs to keep servers running when basement pumps failed.
• The reliance on heroic individual effort to mitigate systemic failures is often viewed as a symptom of poor leadership and inadequate planning, rather than a genuine success.
• Workplace culture can easily drift toward the exploitation of employee dedication, where "heroism" is used as a cheap substitute for hiring sufficient staff or investing in proper redundancy.
• Defining disaster recovery (DR) is complex; while some organizations prioritize cost-saving by accepting infrequent, high-effort manual interventions, others view the lack of robust failover as a fundamental professional failure.
• The common industry practice of lionizing emergency "heroics" can mask deeper issues, such as the failure to anticipate risks or the lack of modern infrastructure like shunt trips for emergency power-offs.
• Maintaining a boundary between personal life and professional obligations is essential, as employers may weaponize an employee's personal circumstances or proximity to the office to demand unpaid, extraordinary labor.
• Decisions regarding data center location and design often involve trade-offs between cost, latency, and environmental risk, and post-hoc criticism of these decisions must account for the limited information available to leaders at the time.
• Fire suppression systems in office-based data centers often rely on sprinklers due to building codes or safety concerns, creating a high-risk environment where a false alarm could necessitate a frantic, manual de-energizing of equipment to prevent water damage.
• Competitive market pressures drive companies to maximize output from their workforce, often resulting in environments that prioritize intensity over long-term stability and employee health.
• The "heroic" response to failures, while impressive at an individual level, should ideally lead to rigorous systemic investigations that ensure such extraordinary measures are never required again.
The discussion reflects a tension between the romanticized view of "heroic" engineering feats and the reality of systemic negligence. While many acknowledge the dedication of individuals who stepped up during disasters like Hurricane Sandy, there is a clear consensus that such efforts often mask chronic underinvestment in basic redundancy and disaster planning. The conversation highlights how corporate culture can exploit personal commitment, emphasizing that while high-pressure scenarios demonstrate individual capability, they simultaneously serve as indicators of failed leadership. Ultimately, the consensus leans toward the idea that true success lies in building resilient systems that do not require superhuman intervention to function during foreseeable crises.