/ Benchmark · 10 Nov 2026

Open models with a rule + Clef
as the lead agent

Pass rate on the lead tasks: initial 29 Oct run, rule + Clef, and Claude Opus 5.5.

Initial, 29 OctRule + ClefClaude Opus 5.5
0%25%50%75%100%53%70%93%GLM 5.335%47%93%GLM Flash56%53%93%Qwen 3.688%72%93%Gemma 483%100%93%DeepSeek
GLM 5.3
Initial53%
Rule + Clef70%
Opus 5.593%
GLM 5.3 Flash
Initial35%
Rule + Clef47%
Opus 5.593%
Qwen 3.6
Initial56%
Rule + Clef53%
Opus 5.593%
Gemma 4
Initial88%
Rule + Clef72%
Opus 5.593%
DeepSeek
Initial83%
Rule + Clef100%
Opus 5.593%
ModelInitial, 29 OctRule + ClefClaude Opus 5.5
GLM 5.3
Z.ai
53% (9 of 17)70% (21 of 30)93% (28 of 30)
GLM 5.3 Flash
Z.ai
35% (6 of 17)47% (14 of 30)93% (28 of 30)
Qwen 3.6
Alibaba Qwen
56% (10 of 18)53% (16 of 30)93% (28 of 30)
Gemma 4
Google
88% (14 of 16)72% (18 of 25)93% (28 of 30)
DeepSeek
DeepSeek
83% (15 of 18)100% (28 of 28, 14 tasks)93% (28 of 30)
  • Three improved: DeepSeek 83% to 100%, GLM 5.3 53% to 70%, GLM 5.3 Flash 35% to 47%.
  • Two did not: Qwen 3.6 56% to 53%, Gemma 4 88% to 72%. Claude Opus 5.5 holds 93%.

Six lead tasks per model; DeepSeek rule + Clef is the 14-task run (28 of 28). Attempts lost to Helmcode errors (HTTP 400/429) are excluded; Opus 5.5 ran on Claude Code, 25 Oct. Small samples.