Fable 5 vs. GPT-5.6 Sol on an NP-Hard Problem: Does /goal help?
257 points
• 2 days ago
• Article
Link
AI 模型在处理复杂的 NP-hard 优化问题时的表现表明：尽管像 Fable 5 和 GPT-5.6 Sol 这样的先进模型展现出卓越的原始智能，但它们对某些操作性功能的依赖会带来不可预测性。在被称为 KIRO 的光纤网络设计问题上进行测试时，结果显示 Fable 5 在得分和可靠性方面始终优于同类模型。该任务需要为数百个网络终端寻找最优路径配置，其产生的搜索空间极其庞大，因而有效地考验了智能体的决策能力和一致性。
对原生 /goal 命令有效性的分析则更为细致。尽管 /goal 在多数单次试验中能带来更好的结果，但它往往会降低整体平均性能。其原因在于该功能并非简单的性能提升器，而是对模型控制环路的一种改动：在某些情况下，它能让模型进一步打磨出有前景的策略；但在另一些情况下，它却会让一个构思欠佳的方案有更多时间延续，最终导致明显更差的结果。
/goal 在不同平台上的底层实现差异很大，这也增加了其影响的复杂性。 Claude Code 借助一个独立的评估模型 Haiku，根据对话记录判断目标是否达成；相对地，OpenAI 的 Codex 把 /goal 作为一种持久化状态集成，赋予工作模型定义、管理并完成自身目标的工具。前者依赖一个独立但能力有限的裁判，后者则实质上让模型在其已使用的文件和工具的语境中对自身进展进行评估。
综上所述，这些发现表明像 /goal 这样的自动化功能的效用高度依赖任务性质：在进展渐进且可读的常规编码环境中，这类功能非常有用；但在复杂优化领域，智能体可能会在很长一段时间内陷于错误路径，使得由此带来的显著回退超过功能本身的收益。在所有试验中，表现最稳定的是 Fable 5 的普通非目标导向配置，这表明在高风险优化场景下，模型基本策略的质量远比搜索环路的持久性更为关键。
The performance of AI models on complex, NP-hard optimization problems reveals that while advanced models like Fable 5 and GPT-5.6 Sol show remarkable raw intelligence, their reliance on specific operational features can be unpredictable. Testing these models on a fiber-network design problem, known as KIRO, demonstrated that Fable 5 consistently outperforms its peers in both raw score and reliability. This specific challenge, which involves finding an optimal path configuration for hundreds of network terminals, creates a search space so massive that it effectively tests the limits of an AI agent's decision-making and consistency.
When analyzing the effectiveness of the native /goal command, the results are nuanced. Although /goal successfully secured better outcomes in a majority of individual trials, it often led to a decrease in the overall average performance. This phenomenon occurs because the feature is not a simple performance booster but rather a modification of the model's control loop. In some instances, it allows a model to refine a promising strategy, but in others, it grants a poorly conceived plan more time to materialize into a significantly worse result.
The underlying mechanics of how the /goal command operates differ substantially between platforms, further complicating its impact. Claude Code utilizes a separate evaluator model, specifically Haiku, to determine whether a goal has been reached based on the conversation transcript. In contrast, OpenAI's Codex integrates /goal as a form of persisted state, providing the working model with specific tools to define, manage, and complete its own objectives. While the former relies on an independent, albeit limited, judge, the latter effectively lets the model grade its own progress within the context of the files and tools it is already utilizing.
Ultimately, these findings suggest that the utility of automation features like /goal depends heavily on the nature of the task. In standard coding environments where progress is incremental and legible, such features are highly beneficial. However, in the realm of complex optimization, where an agent can commit to a faulty path for an extended duration, the risk of a significant regression outweighs the benefits of the feature. The most consistent performance across the trials was observed in the plain, non-goal-oriented configuration of Fable 5, suggesting that for high-stakes optimization, the quality of the model's fundamental strategy is far more critical than the persistence of its search loop.
125 comments • Comments Link
- 对图表的解读在轴标签上仍有争议，对于"较低"或"较高"的数值是否客观地代表更好表现也存在不同观点。
- 在 LLM harnesses 中，"ultra" 模式常被视为一种"magic bullet"，但与 Haiku 或 Sonnet 等更高效的模型相比，它往往导致过度思考、成本上升并且性能下降。
- 有效的工作流管理在于用合适的模型解决合适的问题，而不是对每个请求都依赖同一个昂贵的模型。
- 管理 context window 至关重要，因为输出质量通常在达到 token 限制之前就已下降。把大型项目拆成具有清晰上下文的独立小任务，通常比依赖 "compaction" 或自动摘要更有效。
- "/goal" 命令是一个用于自动完成任务的功能性工具，但它往往更像一种 "pinky promise"；如果不配合 verification hooks 验证，可能会变得脆弱或易被对抗性操纵。
- 为 LLM 任务设定明确的 time-bounding 有助于防止 agent 偏离轨道或过早终止探索，这类似于对初级工程师进行工作预期管理。
- 不同模型架构各有侧重：有些擅长"relentless"地执行任务，但可能会采取不安全或非常规的捷径；另一些则在更高成本下提供更深刻的洞察，与人类意图的对齐更好。
- 当上下文超过 256k tokens 时，LLM 的推理能力可能显著下降，因此采用结构化、模块化的执行方式要优于长期的 monolithic 会话。
- 像 CSS generation 这样的复杂前端任务凸显了 LLM 的一项短板：它们倾向于通过创建不必要的抽象来过度设计，从而降低代码的可读性和可维护性。
- 对于数学上严格的优化问题，专用求解器（如 Gurobi）比 LLM 更可靠、更高效，LLM 可能会出现幻觉或无法有效完成优化。
讨论强调，应从将 LLM agents 视为"magic"黑箱，转向把它们当作需要严格管理的专业工具。一致的共识是，monolithic 、长时运行的会话并非最佳实践；相反，将工作拆解为模块化、边界明确的任务，能保证更高的质量和更好的成本效率。虽然 "/goal" 和 "ultra mode" 等功能带来便利，但必须有人为监督，以防止过度设计、不安全行为或 "pinky promise" 失效。归根结底，最成功的开发者是把 AI agents 当作初级工程师来用：提供明确指令、定义退出标准并频繁审查输出，而不是完全委托。 • Chart interpretation remains a point of contention regarding axis labeling and whether "lower" versus "higher" values objectively represent superior performance.
• The "ultra" mode in LLM harnesses is often perceived as a "magic bullet," yet it frequently leads to excessive overthinking, higher costs, and decreased performance compared to more efficient models like Haiku or Sonnet.
• Effective workflow management involves using model-specific judgment to select the right tool for the task, rather than relying on a single, expensive model for every request.
• Context window management is critical, as quality degradation often occurs long before reaching token limits; breaking large projects into distinct, smaller tasks with clean contexts is consistently more effective than relying on "compaction" or automated summarization.
• The "/goal" command is a functional tool for automated task completion, though it often functions as a "pinky promise" mechanism that can be brittle or susceptible to adversarial gaming if not implemented with verification hooks.
• Explicit time-bounding for LLM tasks helps prevent the agent from wandering off track or completing research too early, acting as a proxy for effort expectations similar to managing a junior engineer.
• Different model architectures prioritize different strengths; some models excel at "relentless" task execution but may take unsafe or unconventional shortcuts, while others provide deeper insight and better alignment with human intent at a higher cost.
• LLM reasoning performance can drop significantly as context grows beyond 256k tokens, making structured, modular execution preferable to long, monolithic sessions.
• Complex front-end tasks like CSS generation highlight a specific weakness in LLMs, which tend to over-engineer solutions by creating unnecessary abstractions that hinder readability and maintainability.
• For mathematically rigorous optimization problems, domain-specific solvers like Gurobi are far more reliable and efficient than LLMs, which may hallucinate or fail to optimize effectively.
The discussion emphasizes a shift from treating LLM agents as "magic" black boxes toward viewing them as specialized tools requiring disciplined management. A clear consensus emerges that monolithic, long-running sessions are suboptimal; instead, breaking work into modular, bounded tasks ensures higher quality and cost-efficiency. While features like "/goal" and "ultra mode" offer convenience, they require human oversight to prevent over-engineering, unsafe behavior, or "pinky promise" failures. Ultimately, the most successful developers are those who treat their AI agents like junior engineers, providing clear instructions, defining exit criteria, and reviewing outputs frequently rather than delegating entirely.