209 comments • Comments Link

长久以来，"pelican on a bicycle" 的 SVG 生成基准测试已成为检验 LLM 能力的一种常见且非正式的"hello world"测试。尽管是否对该题目进行了针对性训练仍有争议，但普遍认为模型也可能只是从整体技术进步中获益。

怀疑者指出，这类基准很可能被训练数据或实验室的目标导向训练所污染；支持者则坚持认为，它仍然是衡量模型审美、构图技巧以及应对新颖且无意义提示词能力的一个有用但并不完美的指标。

当把注意力从该基准迁移到其他同样荒诞且未被广泛基准化的提示词（例如 "a sloth riding a skateboard"）时，模型性能会出现明显且可观测的差异。这表明，即便是最先进的 frontier models，在各种离奇场景中保持一致性仍然困难重重。

通过 SVG 生成来评估模型可以揭示其在 visual reasoning 方面的能力，但模型通常难以自我修正：在被要求检查自己的输出时，经常无法识别出渲染错误。

模型在处理 pelican 和 bicycle 时倾向于默认采用从左向右的运动轨迹，这很可能受阅读方向偏好和摄影中常见的构图法则影响——即主体通常安排成向画面中心或右侧移动。

对来自西方和东方实验室的 proprietary models 进行比较显示，参数数量已不再是衡量智能的可靠代理；attention mechanisms 、 RL tuning 以及架构效率在性能中扮演着愈发重要的角色。

除了静态图像之外，使用 video generation 或基于 SVG 的 animation 等更复杂的任务来测试模型，能更清晰地反映出"品味"和创意规划能力，因为这些任务需要持续的叙事连贯性，而这是简单提示词无法体现的。

这个 benchmark 经常被描述为带有表演性和主观性的实验，但它之所以长期存在，恰恰是因为它占据了一个独特位置：作为一个人类可读、可重复的测试，能够以形式化 benchmark 常常捕捉不到的方式探测模型泛化能力的极限。

关于生成这些 SVG 的成本问题，通常从 developer efficiency 的角度来考量，尽管也有人认为这种视角忽视了更广泛的经济现实——即在性价比上，LLM 已经远超其所替代的人类劳动力。

最终，pelican 项目作为一个社区驱动的产物得以保留，连接了技术评估与数字文化，其持续存在记录了这一行业快速且常常不可预测的发展轨迹。

这场讨论反映了科学界对严谨无偏 benchmark 的追求与用户在与 AI 互动时那种务实且带有戏谑色彩的态度之间的张力。尽管许多参与者承认 "pelican on a bicycle" 测试因在训练数据中过于常见而可能变得陈旧或带有偏见，但他们认为其价值在于作为一个易于理解且非抽象的代理，用来评估模型的"taste"。各方普遍认为，虽然 frontier models 已取得显著进步，但在处理类似荒诞且未经优化的提示词时仍会持续失败，这凸显了当前 AI 能力更多依赖记忆与模式复制，而非真正广义的创造性推理。 • The long-running "pelican on a bicycle" SVG generation benchmark has become a common, informal "hello world" test for LLM capabilities, despite ongoing debates regarding whether models are intentionally trained to solve it or simply benefiting from general advancements.

• While skeptics argue the benchmark is likely polluted by training data and potential target-training by labs, others maintain that it serves as a useful, albeit imperfect, indicator of a model's "taste," composition skills, and ability to handle novel, nonsensical prompts.

• There is a notable, observable delta in model performance when moving from this benchmark to other equally absurd, unbenchmarked prompts like "a sloth riding a skateboard," suggesting that consistency across varying outlandish scenarios remains an elusive goal for even the most advanced frontier models.

• Evaluating model performance via SVG generation offers insights into "visual reasoning," yet models often struggle with self-correction, frequently failing to identify their own rendering errors when asked to review their output.

• The persistent trend of models defaulting to a left-to-right motion for the pelican and bicycle is likely influenced by reading direction biases and standard compositional rules in photography, which dictate that subjects should be framed to move toward the center or right of the frame.

• Comparisons between proprietary models from Western and Eastern labs highlight that parameter count is no longer a reliable proxy for intelligence, with attention mechanisms, RL tuning, and architectural efficiency playing increasingly critical roles in performance.

• Beyond static images, testing models with complex tasks like video generation or SVG-based animation provides a clearer picture of "taste" and creative planning, as these tasks require sustained narrative coherence that simpler prompts may mask.

• The "benchmark" is frequently characterized as a performative and subjective experiment, yet it persists precisely because it occupies a unique space as a human-readable, repeatable test that probes the limits of model generalization in a way that formal benchmarks often fail to capture.

• Concerns regarding the cost of generating these SVGs are often framed through the lens of developer efficiency, though some argue this perspective ignores the broader economic reality where LLMs are already vastly more cost-effective than the human labor they replace.

• Ultimately, the pelican project survives as a community-driven artifact that bridges technical evaluation and digital culture, with its ongoing existence serving as a record of the industry's rapid, often unpredictable, progression.

The conversation reflects a tension between the scientific desire for rigorous, unbiased benchmarking and the pragmatic, often playful, way users actually interact with AI. While many participants acknowledge that the "pelican on a bicycle" test has become potentially stale or biased due to its popularity in training data, they find its value lies in its ability to serve as a relatable, non-abstract proxy for evaluating model "taste." There is a clear consensus that frontier models have made significant strides, yet the persistent failure to handle similarly absurd, unoptimized prompts underscores that current AI capabilities often rely more on memorization and pattern replication than true, generalized creative reasoning.