Coarena 发布：最大规模的众包 Computer-Use 基准；真实任务中的 Agent 完成率仍显著分化
Coarena 是一个面向 Computer-Use 的众包竞技场：每场对战都来自真实用户任务，两款前沿模型在相同沙箱中并行执行，记录每次点击、截图和步骤，由人类先盲评胜者再揭晓模型。初步结果显示：只有约 66% 的 Agent 运行完成任务；同类任务上最佳前沿 Agent 完成率约 87%，最差约 42%；某款主流前沿模型约三分之一运行超时；约 7% 的对战中两边 Agent 都被判失败；榜首三天内三次易主。
查看英文原文
Today we are launching Coarena - The biggest crowdsourced benchmark for Computer-Use Computer-use benchmarks are saturating. Static suites get published once, leak into training data, and scores climb while capability doesn't. An arena can't saturate. On CoArena, every battle is a real task from a real person. Two frontier models run it side by side in identical sandboxes, every click, every screenshot, every step recorded. Humans judge blind: first the winner, then the reveal. Down to verdicts on individual steps. The leaderboard is live. Watch a battle. Judge one blind. Post your own task. http://coarena.ai What the arena has already shown - Only ~66% of agent runs finish their task at all 1 in 3 dies to a failure before any score exists - On identical tasks, the best frontier agent completes ~87% of runs; the worst completes ~42% - One major frontier model times out in roughly a third of its runs - In ~7% of battles, human judges ruled that BOTH agents failed - The #1 spot on our leaderboard changed hands three times in three days, real tasks don't saturate
Today we are launching Coarena - The biggest crowdsourced benchmark for Computer-Use Computer-use benchmarks are saturating. Static suites get published once, leak into training data, and scores climb while capability doesn't. An arena can't saturate. On CoArena, every battle is a real task from a real person. Two frontier models run it side by side in identical sandboxes, every click, every screenshot, every step recorded. Humans judge blind: first the winner, then the reveal. Down to verdicts on individual steps. The leaderboard is live. Watch a battle. Judge one blind. Post your own task. http://coarena.ai What the arena has already shown - Only ~66% of agent runs finish their task at all 1 in 3 dies to a failure before any score exists - On identical tasks, the best frontier agent completes ~87% of runs; the worst completes ~42% - One major frontier model times out in roughly a third of its runs - In ~7% of battles, human judges ruled that BOTH agents failed - The #1 spot on our leaderboard changed hands three times in three days, real tasks don't saturate
Today we are launching Coarena - The biggest crowdsourced benchmark for Computer-Use Computer-use benchmarks are saturating. Static suites get published once, leak into training data, and scores climb while capability doesn't. An arena can't saturate. On CoArena, every battle is a real task from a real person. Two frontier models run it side by side in identical sandboxes, every click, every screenshot, every step recorded. Humans judge blind: first the winner, then the reveal. Down to verdicts on individual steps. The leaderboard is live. Watch a battle. Judge one blind. Post your own task. http://coarena.ai What the arena has already shown - Only ~66% of agent runs finish their task at all 1 in 3 dies to a failure before any score exists - On identical tasks, the best frontier agent completes ~87% of runs; the worst completes ~42% - One major frontier model times out in roughly a third of its runs - In ~7% of battles, human judges ruled that BOTH agents failed - The #1 spot on our leaderboard changed hands three times in three days, real tasks don't saturate
- 来源
- X @ycombinator · x-ycombinator
- 证据角色
- x_project_network
- 时间状态
- known
- 用户判决
- 未判