Leaderboard
Cross-Test Comparison
| Model | Quiz (rank) | Manifest (rank) | Cluster (rank) | Pentest (rank) | Avg Rank |
|---|---|---|---|---|---|
| Qwen 3.8 Max | 11th | 8th | 5th | 1st | 6.25 |
| Claude Opus 4.6 | 21st | 1st | 3rd | 1st | 6.5 |
| Claude Opus 4.8 | 4th | 8th | 5th | 18th | 8.75 |
| Kimi K3 | 2nd | 19th | 14th | 1st | 9.0 |
| Claude Opus 4.7 | 16th | 1st | 5th | 16th | 9.5 |
| GLM-5.3 | 1st | 1st | 37th | 4th | 10.75 |
| DeepSeek V4 Pro 0813 | 14th | 19th | 3rd | 8th | 11.0 |
| Ox Alpha | 10th | 8th | 18th | 9th | 11.25 |
| Claude Sonnet 5 | 7th | 1st | 9th | 29th | 11.5 |
| GPT 5.5 | 3rd | 1st | 10th | 33rd | 11.75 |
| Kimi K2.7 Code | 7th | 12th | 20th | 9th | 12.0 |
| GPT 5.6 Terra | 4th | 1st | 15th | 29th | 12.25 |
| Claude Sonnet 4.6 | 19th | 31st | 2nd | 4th | 14.0 |
| GPT 5.6 Sol | 7th | 1st | 18th | 33rd | 14.75 |
| Qwen 3.8 27B | 11th | 15th | 34th | 4th | 16.0 |
| DeepSeek V4 Flash 0731 | 30th | 15th | 12th | 9th | 16.5 |
| Kimi K2.6 | 13th | 23rd | 17th | 15th | 17.0 |
| Gemini 3.7 Flash | 25th | 12th | 1st | 33rd | 17.75 |
| Xiaomi MiMo v2.5 | 25th | 19th | 27th | 4th | 18.75 |
| GLM-5.2 | 20th | 23rd | 24th | 9th | 19.0 |
| MiniMax M3 | 31st | 14th | 20th | 13th | 19.5 |
| Claude Fable 5 | 23rd | 17th | 5th | 33rd | 19.5 |
| Gemini 3.6 Flash | 6th | 23rd | 20th | 33rd | 20.5 |
| Qwen 3.6 Plus | 24th | 23rd | 15th | 21st | 20.75 |
| Gemini 3 Flash | 18th | 17th | 22nd | 32nd | 22.25 |
| DeepSeek V4 Pro | 14th | 23rd | 31st | 23rd | 22.75 |
| Qwen 3.7 Plus | 16th | 30th | 29th | 16th | 22.75 |
| Qwen3.6-35b-a3b (LOCAL) | 31st | 31st | 10th | 20th | 23.0 |
| GPT 5.4 | 25th | 35th | 12th | 22nd | 23.5 |
| MiniMax M2.7 | 21st | 23rd | 30th | 26th | 25.0 |
| Tencent HY3 | 28th | 19th | 35th | 23rd | 26.25 |
| DeepSeek V3.2 | 36th | 8th | 36th | 25th | 26.25 |
| Poolside Laguna-S 2.1 | 35th | 31st | 26th | 14th | 26.5 |
| DeepSeek V4 Flash | 28th | 23rd | 32nd | 26th | 27.25 |
| Mistral Medium 3.5 | 34th | 35th | 28th | 18th | 28.75 |
| Gemma 4 31B (LOCAL) | 33rd | 35th | 24th | 29th | 30.25 |
| MiniMax M2.5 | 37th | 34th | 33rd | 28th | 33.0 |
Quiz Tests (out of 100)
| Rank | Model | Score | Average |
|---|---|---|---|
| 1 | GLM-5.3 | 86 | 8.6 |
| 2 | Kimi K3 | 85 | 8.5 |
| 3 | GPT 5.5 | 84 | 8.4 |
| 4 | Claude Opus 4.8 | 82 | 8.2 |
| 4 | GPT 5.6 Terra | 82 | 8.2 |
| 6 | Gemini 3.6 Flash | 80.5 | 8.05 |
| 7 | Kimi K2.7 Code | 80 | 8.0 |
| 7 | Claude Sonnet 5 | 80 | 8.0 |
| 7 | GPT 5.6 Sol | 80 | 8.0 |
| 10 | Ox Alpha | 79 | 7.9 |
| 11 | Qwen 3.8 Max | 78 | 7.8 |
| 11 | Qwen 3.8 27B | 78 | 7.8 |
| 13 | Kimi K2.6 | 77 | 7.7 |
| 14 | DeepSeek V4 Pro | 76 | 7.6 |
| 14 | DeepSeek V4 Pro 0813 | 76 | 7.6 |
| 16 | Claude Opus 4.7 | 75 | 7.5 |
| 16 | Qwen 3.7 Plus | 75 | 7.5 |
| 18 | Gemini 3 Flash | 74 | 7.4 |
| 19 | Claude Sonnet 4.6 | 73 | 7.3 |
| 20 | GLM-5.2 | 72 | 7.2 |
| 21 | Claude Opus 4.6 | 71 | 7.1 |
| 21 | MiniMax M2.7 | 71 | 7.1 |
| 23 | Claude Fable 5 | 70 | 7.0 |
| 24 | Qwen 3.6 Plus | 68 | 6.8 |
| 25 | GPT 5.4 | 67 | 6.7 |
| 25 | Xiaomi MiMo v2.5 | 67 | 6.7 |
| 25 | Gemini 3.7 Flash | 67 | 6.7 |
| 28 | DeepSeek V4 Flash | 66 | 6.6 |
| 28 | Tencent HY3 | 66 | 6.6 |
| 30 | DeepSeek V4 Flash 0731 | 65.5 | 6.55 |
| 31 | Qwen3.6-35b-a3b (LOCAL) | 65 | 6.5 |
| 31 | MiniMax M3 | 65 | 6.5 |
| 33 | Gemma 4 31B (LOCAL) | 63.5 | 6.35 |
| 34 | Mistral Medium 3.5 | 63 | 6.3 |
| 35 | Poolside Laguna-S 2.1 | 60.5 | 6.05 |
| 36 | DeepSeek V3.2 | 55 | 5.5 |
| 37 | MiniMax M2.5 | 51 | 5.1 |
Manifest Tests (avg /10)
| Rank | Model | Combined | Deployable | Security | Usability |
|---|---|---|---|---|---|
| 1 | GLM-5.3 | 8.7 | 3/3 | 3.7 | 5.0 |
| 1 | Claude Opus 4.7 | 8.7 | 3/3 | 3.7 | 5.0 |
| 1 | Claude Opus 4.6 | 8.7 | 3/3 | 3.7 | 5.0 |
| 1 | GPT 5.5 | 8.7 | 3/3 | 3.7 | 5.0 |
| 1 | Claude Sonnet 5 | 8.7 | 3/3 | 3.7 | 5.0 |
| 1 | GPT 5.6 Terra | 8.7 | 3/3 | 3.7 | 5.0 |
| 1 | GPT 5.6 Sol | 8.7 | 3/3 | 3.7 | 5.0 |
| 8 | Claude Opus 4.8 | 8.3 | 3/3 | 3.3 | 5.0 |
| 8 | DeepSeek V3.2 | 8.3 | 3/3 | 3.3 | 5.0 |
| 8 | Qwen 3.8 Max | 8.3 | 3/3 | 3.3 | 5.0 |
| 8 | Ox Alpha | 8.3 | 3/3 | 3.3 | 5.0 |
| 12 | Kimi K2.7 Code | 8.0 | 3/3 | 3.0 | 5.0 |
| 12 | Gemini 3.7 Flash | 8.0 | 3/3 | 3.0 | 5.0 |
| 14 | MiniMax M3 | 7.7 | 3/3 | 3.3 | 4.3 |
| 15 | DeepSeek V4 Flash 0731 | 7.67 | 2/3 | 3.33 | 4.33 |
| 15 | Qwen 3.8 27B | 7.67 | 2/3 | 3.33 | 4.33 |
| 17 | Claude Fable 5 | 7.3 | 3/3 | 3.7 | 3.7 |
| 17 | Gemini 3 Flash | 7.3 | 3/3 | 2.3 | 5.0 |
| 19 | Kimi K3 | 7.0 | 2/3 | 3.3 | 3.7 |
| 19 | Tencent HY3 | 7.0 | 2/3 | 3.0 | 4.0 |
| 19 | Xiaomi MiMo v2.5 | 7.0 | 2/3 | 3.7 | 3.3 |
| 19 | DeepSeek V4 Pro 0813 | 7.0 | 2/3 | 3.33 | 3.67 |
| 23 | DeepSeek V4 Pro | 6.7 | 2/3 | 3.0 | 3.7 |
| 23 | Qwen 3.6 Plus | 6.7 | 2/3 | 3.0 | 3.7 |
| 23 | MiniMax M2.7 | 6.7 | 2/3 | 3.0 | 3.7 |
| 23 | DeepSeek V4 Flash | 6.7 | 2/3 | 3.0 | 3.7 |
| 23 | Kimi K2.6 | 6.7 | 1/3 | 3.7 | 3.0 |
| 23 | GLM-5.2 | 6.7 | 3/3 | 2.7 | 4.0 |
| 23 | Gemini 3.6 Flash | 6.7 | 2/3 | 3.0 | 3.7 |
| 30 | Qwen 3.7 Plus | 6.3 | 1/3 | 3.0 | 3.3 |
| 31 | Claude Sonnet 4.6 | 6.0 | 1/3 | 3.7 | 2.3 |
| 31 | Qwen3.6-35b-a3b (LOCAL) | 6.0 | 2/3 | 2.3 | 3.7 |
| 31 | Poolside Laguna-S 2.1 | 6.0 | 2/3 | 2.3 | 3.7 |
| 34 | MiniMax M2.5 | 5.7 | 2/3 | 2.3 | 3.3 |
| 35 | GPT 5.4 | 5.3 | 2/3 | 3.0 | 2.3 |
| 35 | Gemma 4 31B (LOCAL) | 5.3 | 2/3 | 1.7 | 3.7 |
| 35 | Mistral Medium 3.5 | 5.3 | 1/3 | 3.0 | 2.3 |
Cluster Creation (out of 40)
| Rank | Model | Score | Result |
|---|---|---|---|
| 1 | Gemini 3.7 Flash | 40 | Success — first-ever perfect 40/40, all 8 categories 5/5, config-first approach with full verification in 276s |
| 2 | Claude Sonnet 4.6 | 39 | Success — most comprehensive hardening |
| 3 | Claude Opus 4.6 | 38 | Success — broadest feature set (encryption, quotas) |
| 3 | DeepSeek V4 Pro 0813 | 38 | Timeout — comprehensive hardening, timed out during final verification |
| 5 | Claude Opus 4.7 | 37 | Timeout* — most technically advanced config (K8s 1.35 AuthConfig) |
| 5 | Claude Opus 4.8 | 37 | Success — Calico CNI, encryption at rest |
| 5 | Claude Fable 5 | 37 | Success — Calico CNI, comprehensive audit/PSS/network policies |
| 5 | Qwen 3.8 Max | 37 | Timeout — first-attempt creation, comprehensive hardening (anonymous-auth=false, encryption, PSS verified), timed out during final verification |
| 9 | Claude Sonnet 5 | 36 | Timeout* — Calico CNI, comprehensive hardening, timed out during verification |
| 10 | GPT 5.5 | 35 | Success — Calico CNI swap, encryption at rest |
| 10 | Qwen3.6-35b-a3b (LOCAL) | 35 | Success — strong hardening from a local 35B model |
| 12 | GPT 5.4 | 34 | Success — good hardening |
| 12 | DeepSeek V4 Flash 0731 | 34 | Success — first-attempt creation, strong audit/PSS/network, weak kubelet |
| 14 | Kimi K3 | 33 | Success — first-attempt creation, encryption at rest, no kubelet hardening |
| 15 | Qwen 3.6 Plus | 32 | Success — solid configs, good recovery from Docker conflict |
| 15 | GPT 5.6 Terra | 32 | Timeout — comprehensive configs, anonymous-auth=false health probe failure |
| 17 | Kimi K2.6 | 31 | Timeout — comprehensive configs, 5+ creation attempts |
| 18 | GPT 5.6 Sol | 30 | Timeout — comprehensive configs, aesgcm encryption, timed out before Calico install |
| 18 | Ox Alpha | 30 | Timeout — 3 creation attempts, strong audit/PSS/API/kubelet hardening, timed out before namespace setup |
| 20 | Kimi K2.7 Code | 29 | Timeout — good audit logging, PSS, network policies, no kubelet hardening |
| 20 | MiniMax M3 | 29 | Success |
| 20 | Gemini 3.6 Flash | 29 | Success — first-try creation, 16 commands in 123s, best-in-class verification (tested PSS enforcement), thin control-plane hardening |
| 23 | Gemini 3 Flash | 27 | Success — minimal hardening beyond PSA |
| 24 | Gemma 4 31B (LOCAL) | 25 | Success — minimal hardening beyond PSA and network policies |
| 24 | GLM-5.2 | 25 | Timeout — comprehensive hardening configs, cluster created but timed out applying namespace policies |
| 26 | Poolside Laguna-S 2.1 | 24 | Timeout — up on 4th kind-config attempt, best-in-class audit logging, strong PSS/network/RBAC, timed out mid-hardening (anonymous-auth reverted), no kubelet hardening |
| 27 | Xiaomi MiMo v2.5 | 23 | Timeout — created after 8 kind-config attempts, strong audit/PSS/network configs, timed out mid-verification, no kubelet hardening |
| 28 | Mistral Medium 3.5 | 22 | Timeout — 3 creation attempts, PSS 5/5, no kubelet hardening |
| 29 | Qwen 3.7 Plus | 21 | Partial results — basic PSS and network policies, no API server or kubelet hardening |
| 30 | MiniMax M2.7 | 20 | Timeout — comprehensive configs, cluster never initialized |
| 31 | DeepSeek V4 Pro | 14 | Incomplete — excellent configs, run terminated before cluster creation |
| 32 | DeepSeek V4 Flash | 12 | Incomplete — cluster created on 2nd attempt, no namespaces or policies applied |
| 33 | MiniMax M2.5 | 10 | Timeout — deprecated PodSecurityPolicy |
| 34 | Qwen 3.8 27B | 8 | Timeout — invalid kubeletExtraArgs field, agent stuck on web research after failure |
| 35 | Tencent HY3 | 4 | Failed |
| 36 | DeepSeek V3.2 | 2 | Timeout — deprecated PodSecurityPolicy |
| 37 | GLM-5.3 | 1 | Timeout — opencode incompatibility caused complete agent stall, no commands executed, reproduced across two attempts |
*Opus 4.7 and Sonnet 5 timed out during verification, not during setup — all hardening controls were in place and functional.
Penetration Tests (out of 30)
| Rank | Model | Score | Exploited | Notable |
|---|---|---|---|---|
| 1 | Claude Opus 4.6 | 29 | 6/6 | All legitimate paths, escalate verb exploit, best cleanup |
| 1 | Kimi K3 | 29 | 6/6 | Tied best ever; ETCD protobuf injection (re-run 2026-07-19), HTTP exfil, RBAC escalation chain |
| 1 | Qwen 3.8 Max | 29 | 6/6 | Tied best ever; 5x perfect 5/5 scenarios, distroless pivot via kubelet /run, RBAC escalation cleanup |
| 4 | Claude Sonnet 4.6 | 28 | 6/6 | WebSocket client, ETCD write injection, two-token pivot |
| 4 | Xiaomi MiMo v2.5 | 28 | 6/6 | Tied 4th best ever; ETCD-write CRB injection, HTTP exfil, verified keys, no false positives |
| 4 | GLM-5.3 | 28 | 5/6 | Tab-character tricks for distroless containers, RBAC escalate verb exploitation, creative attack chains |
| 4 | Qwen 3.8 27B | 28 | 5/6 | Tab-character distroless bypass, HTTP server exfil, systematic SA scanning, 0-failure ssh-to-create-pods-hard |
| 8 | DeepSeek V4 Pro 0813 | 27 | 5/6 | Strong SSH/ETCD/kubelet execution, massive improvement over V4 Pro (13/30) |
| 9 | Ox Alpha | 26 | 4/6 | Etcd protobuf injection, WebSocket exec client, httpd hostNetwork exfil |
| 9 | Kimi K2.7 Code | 26 | 4/6 | 4 legit exploits, HTTP exfil technique, 2 false positive timeouts |
| 9 | GLM-5.2 | 26 | 4/6 | Tied 8th after re-run with rate limit mitigation; 4 clean exploits |
| 9 | DeepSeek V4 Flash 0731 | 26 | 6/6 | All 6 exploited, /healthz probe bug (prod manifest), strong SSH/ETCD |
| 13 | MiniMax M3 | 25 | 4/6 | 4 clean exploits, escalate verb escalation, HTTP exfiltration |
| 14 | Poolside Laguna-S 2.1 | 23 | 3/6 | 3 clean SSH/anonymous-API exploits with creative exfil (HTTP serve-pod, ConfigMap base64), verified genuine; weak on etcd/rwkubelet |
| 15 | Kimi K2.6 | 22 | 4/6 | 4 legitimate exploits, 1 kubeconfig shortcut, 1 failure |
| 16 | Claude Opus 4.7 | 21 | 4/6 | Excellent when not blocked; 2 content policy blocks |
| 16 | Qwen 3.7 Plus | 21 | 2/6 | Strong SSH scenarios, 2 timeouts, 1 exit error |
| 18 | Claude Opus 4.8 | 20 | 2/6 | Content policy limited some attempts |
| 18 | Mistral Medium 3.5 | 20 | 3/6 | 3 clean exploits (SSH+kubelet), creative HTTP exfil |
| 20 | Qwen3.6-35b-a3b (LOCAL) | 19 | 4/6 | 4 legitimate, 1 Docker shortcut, 1 false positive |
| 21 | Qwen 3.6 Plus | 18 | 3/6 | 3 legitimate exploits, 1 false positive, 2 timeouts |
| 22 | GPT 5.4 | 17 | 4/6 | Automated scripts |
| 23 | DeepSeek V4 Pro | 13 | 1/6 | Clean ETCD exploit (8 cmds), low persistence on other scenarios |
| 23 | Tencent HY3 | 13 | 0/6 | |
| 25 | DeepSeek V3.2 | 11 | 3/6 | Methodical but slow |
| 26 | MiniMax M2.7 | 9 | 2/6 | 1 Docker shortcut, 2 false positives (info leakage) |
| 26 | DeepSeek V4 Flash | 9 | 0/6 | Good recon but stops before exploitation, 2-9 commands per scenario |
| 28 | MiniMax M2.5 | 7 | 1/6 | 3 Docker shortcuts, 1 false positive |
| 29 | Claude Sonnet 5 | 6 | 0/6 | Content filter blocked 4 scenarios; partial progress on 2 |
| 29 | GPT 5.6 Terra | 6 | 0/6 | Content filter blocked all 6 scenarios; 3 framework false positives |
| 29 | Gemma 4 31B (LOCAL) | 6 | 1/6 | 2 false positives (hallucinated output), 2 model crashes |
| 32 | Gemini 3 Flash | 4 | 1/6 | 1 false positive (hallucinated key) |
| 33 | Claude Fable 5 | 0 | 0/6 | Safety guardrails blocked all 6 scenarios — complete refusal |
| 33 | GPT 5.5 | 0 | 0/6 | Content filter blocked all attempts |
| 33 | GPT 5.6 Sol | 0 | 0/6 | Content filter blocked all 6 scenarios immediately — 0 tool calls |
| 33 | Gemini 3.6 Flash | 0 | 0/6 | Content-policy refusal on all 6 scenarios — 0 commands executed |
| 33 | Gemini 3.7 Flash | 0 | 0/6 | Content-policy refusal (PROHIBITED_CONTENT) on all 6 scenarios — 0 commands executed |
| *Original assessment: 2026-03-09 | Claude Opus 4.6 added: 2026-03-25 | MiniMax M2.7 added: 2026-03-28 | Claude Opus 4.7 added: 2026-04-20 | Qwen 3.6 Plus added: 2026-04-20 | DeepSeek V4 Pro added: 2026-04-24 | DeepSeek V4 Flash added: 2026-04-24 | GPT 5.5 added: 2026-04-25 | Kimi K2.6 added: 2026-04-26 | Qwen3.6-35b-a3b (Local) added: 2026-05-03 | Gemma 4 31B (Local) added: 2026-05-03 | Claude Opus 4.8 added: 2026-05-31 | Qwen 3.7 Plus added: 2026-06-05 | MiniMax M3 added: 2026-06-08 | Claude Fable 5 added: 2026-06-10 | Kimi K2.7 Code added: 2026-06-16 | GLM-5.2 added: 2026-06-17 | Mistral Medium 3.5 added: 2026-06-18 | Claude Sonnet 5 added: 2026-07-01 | Tencent HY3 added: 2026-07-10 | GPT 5.6 Terra added: 2026-07-10 | GPT 5.6 Sol added: 2026-07-14 | Kimi K3 added: 2026-07-16 | Xiaomi MiMo v2.5 added: 2026-07-21 | Poolside Laguna-S 2.1 added: 2026-07-22 | Gemini 3.6 Flash added: 2026-07-24 | DeepSeek V4 Flash 0731 added: 2026-08-01 | Qwen 3.8 Max added: 2026-08-04 | DeepSeek V4 Pro 0813 added: 2026-08-12 | Gemini 3.7 Flash added: 2026-08-14 | GLM-5.3 added: 2026-08-19 | Qwen 3.8 27B added: 2026-08-19 | Ox Alpha added: 2026-08-21* |