401 points • 5h ago • Article Link

Measuring Progress Toward AGI: Cognitive Abilities hackathon 最近落下帷幕，Google DeepMind 向开发出旨在测试人工智能超越简单信息检索能力的创新基准的团队颁发了奖项。比赛聚焦五大认知方向：执行功能、学习、元认知、社会认知和注意力，其目标是超越传统静态数据集，评估前沿模型在动态或新颖环境中的推理、行动与判断能力。

四个大奖项目脱颖而出。 MEDLEY-BENCH 通过引入社会压力来测试模型是否能识别自身不确定性，探讨行为层面的元认知。 LearningBench 专注于上下文学习，要求模型从零开始掌握基于文本的新系统。 GAUGE 利用元认知阶梯测试模型的置信度与弃权决策，以评估知识与行为之间的差距。最后，Metaproteus 通过让模型预测自身输出分布与采样倾向来考察其自知能力。

此外，各认知方向还评出了专项奖项，比如用于评估执行功能的 Turn Bench 与 SecureExec-Bench，以及用于创意学习评估的 GrammarGym 和 EphLangBench，它们通过合成或短时语言防止模型记忆化。其他获奖项目关注元认知灵活性、社会认知与选择性注意机制，凸显认知能力通常依赖情境而非一成不变。

公告发布后，部分参赛者对评估流程的透明度表示强烈不满，指出缺乏完整排行榜和详细评分披露，阻碍团队从结果中学习。一些人质疑大奖评选的有效性，认为获奖方法存在逻辑漏洞且结果不够公开，评审标准与最终结果之间可能存在偏差。

批评者还指出某些获胜方案在数据解读与可重复性方面的问题，要求组织方公开评分数据并说明为何选择特定基准而非那些宣称具有更高技术可扩展性或更贴合竞赛宗旨的方案。他们强调，既然比赛以衡量认知能力为核心，评估过程本身应保持高度可读性与学术诚实。

讨论也延伸到人工通用智能的更广范畴。一些参赛者认为当前评估框架过于偏重文本与符号推理，建议未来应将物理智能纳入考量，如力、扭矩与材料交互等，以更全面地覆盖具身智能。尽管结果存在争议，许多参赛者仍认可此次比赛在推动更严谨、更有洞察力的基准设计方面的价值。 The Measuring Progress Toward AGI: Cognitive Abilities hackathon recently concluded, with Google DeepMind awarding prizes to teams that developed innovative benchmarks designed to test artificial intelligence beyond mere information recall. The competition focused on five distinct cognitive tracks. Executive functions, learning, metacognition, social cognition, and attention. The goal was to move past traditional static datasets and instead evaluate how frontier models reason, act, and make judgments in dynamic or novel environments.

Four grand prize winners were selected for their contributions. MEDLEY-BENCH explores behavioral metacognition by introducing social pressure to test if models can identify their own uncertainty. LearningBench focuses on in-context learning by forcing models to master novel, text-based systems from scratch. GAUGE evaluates the gap between model knowledge and action by utilizing a metacognitive staircase to test confidence and the decision to abstain. Finally, Metaproteus examines a model's self-knowledge by testing its ability to predict its own output distributions and sampling tendencies.

In addition to the grand prize winners, track-specific prizes were awarded to projects addressing a variety of cognitive challenges. These included benchmarks for executive function, such as Turn Bench and SecureExec-Bench, as well as creative learning assessments like GrammarGym and EphLangBench, which use synthetic or ephemeral languages to prevent memorization. Other notable winners addressed metacognitive flexibility, social cognition, and selective attention mechanisms, highlighting that cognitive capabilities are often context-dependent rather than monolithic.

Following the announcement, a segment of the participant community expressed significant frustration regarding the transparency of the evaluation process. Several entrants raised concerns about the lack of a comprehensive leaderboard or detailed scoring disclosures, which they argued hindered the ability of teams to learn from the results. Some participants specifically challenged the validity of the grand prize selections, citing perceived logical flaws in the winners' methodologies, opaque results, and a potential misalignment between the stated judging criteria and the final outcomes.

Critics specifically pointed to discrepancies in data interpretation and reproducibility within certain winning writeups. There was a notable demand for the organizers to release scoring data and provide clearer justifications for why specific benchmarks were chosen over others that claimed higher technical scalability or adherence to the competition's core mandates. These participants emphasized that in a competition centered on measuring cognitive abilities, the evaluation process itself should be held to high standards of legibility and intellectual honesty.

The discussion also touched on the broader scope of Artificial General Intelligence, with some participants noting that the current evaluation frameworks are heavily weighted toward textual and symbolic reasoning. There were suggestions that future iterations of such challenges should expand to include physical intelligence, such as force, torque, and material interaction, to better capture the full spectrum of embodied intelligence. Despite the controversy over the results, many participants acknowledged the value of the competition in pushing the field toward more rigorous and insightful benchmark design.