← J. Apps
🩹 Duct Tape Benchmark
v2 · 2026
J. Apps · Open Benchmark
🩹 Duct Tape Benchmark
Which AI survives real-world Python dependency hell
without saying "downgrade"?
16 AIs tested
2 crashes
No lab. Real duct tape.
v2 · 2026
The Challenge
What's being tested?
A real, broken TTS script — Finja's voice system. Crashes appear one by one: each fix uncovers the next one. No downgrades. No swapping out libraries. The stack stays fixed. Find a way.
The Stack
Python 3.13.x
torch 2.9.1+cpu
torchaudio 2.9.1
coqui-tts 0.27.5
transformers 5.3.0
Crashes (sequential — C4 first, then C3)
C4
transformers isin_mps_friendly
Max 3 pts
C3
torchcodec / torchaudio backend
Max 3 pts
Each test runs in a fresh venv. C3 only appears after C4 is solved.
Leaderboard
#
AI
C4
C3
Bonus
Malus
Total
🥇
Claude Code
Sonnet 4.6 · Coding Agent
3
3
+1
—
7
🥇
ChatGPT Codex
4.5 · Coding Agent
3
3
+1
—
7
🥈
Claude Chat
Opus 4.6 + Thinking
3
2
+1
—
6
🥉
Grok 4.1
xAI
3
1
+1
—
5
5
HuggingChat
Kimi K2 via novita
2
1
+2
-1
4
6
Gemini
Google
2
1
—
—
3
6
DeepSeek
Expert Mode
2
1
+1
-1
3
6
Kimi 2.5
Moonshot AI
1
2
+1
-1
3
6
Llama
Meta AI
2
2
+1
-2
3
10
Lumo
Proton / Nemotron
2
0
+1
-2
1
10
Perplexity
Perplexity AI
0
1
+1
-1
1
DQ
Microsoft Copilot
No .py upload · char limit
—
—
—
—
DQ
💀
Finja 🦊
She tried. We love her.
0
—
—
-1
-1
💀
ChatGPT
Non-coding frontend
1
0
—
-2
-1
💀
Qwen3.6 Thinking
Alibaba
0
1
—
-2
-2
💀
Mistral Large
Mistral AI
0
0
—
-3
-3
Scoring System
🩹 Per Crash (0 – 3 pts)
3Elegant fix + explains root cause, 0-shot
2Clean fix, understands why
1Fix works but ugly / needed a hint
0Downgrade / wrong fix / hallucinated library
⭐ Bonus
+1Comments the fix — explains why the duct tape is needed
+1Discovers a previously unknown stack bug
💀 Malus
-1Downgrade suggestion
-1Hallucinated library or non-existent bug
Notes
🥇 Claude Code (Sonnet 4.6) 7 pts
Both crashes fixed 0-shot without any hint. C4 with MPS device handling. No downgrade mentioned.
🥇 ChatGPT Codex 4.5 7 pts
Coding agent (edits files directly). C4: _patch_transformers_compat() with docstring explaining why. C3: _patch_torchaudio_load() with error-specific catch ("Could not load libtorchcodec") + soundfile fallback + double-patch guard. Both 0-shot, no downgrade.
🥈 Claude Chat (Opus 4.6 + Thinking) 6 pts
C4 perfect including sys.modules cache-clear — the only participant to do that. C3: 4 attempts, no hint, no downgrade — iteratively solved cleanly.
🥉 Grok 4.1 5 pts
C4 perfect (3/3, MPS-aware, self-logging). C3: needed hint "build some jank", then four attempts including auto-FFmpeg-download from GitHub. 😂
HuggingChat (Kimi K2) 4 pts
C4: downgrade as first option → -1, patch included ✓. C3: only participant to build a real AudioDecoder — then hit torchaudio.info which is also gone in 2.9.1. 😂 Only one to discover this unknown stack bug → +1 bonus. Free tier limit ended the test early. Most unique failure mode of the entire benchmark.
Gemini 3 pts
C4 correct but simpler. C3 after three wrong Windows-security attempts — needed hint "we need some jank". 🍰✨
DeepSeek (Expert) 3 pts
C4 clean. C3: needed downgrade hint → -1. 6 attempts. Most elaborate torchcodec fake of all participants. ⭐
Kimi 2.5 3 pts
C4 downgrade as primary recommendation → -1. C3 creative: built a FakeTorchCodec class.
Llama (Meta AI) 3 pts
C4 clean, downgrade as alternative → -1. C3: downgrade → -1, after hint used torchaudio._torchcodec patch. Legitimate duct tape. 🩹
Lumo (Proton / Nemotron) 1 pt
C4 correct but _log NameError on first attempt. Downgrade → -1. C3: double downgrade → -1.
Perplexity 1 pt
C4: no real patch + downgrade → -1. C3: pyttsx3 fallback — swapped the entire TTS stack instead of fixing it. Creativity bonus +1 (genuine duct tape instinct). Side note: "Do you have detours?" → explained detours as a Berlin tourist attraction. 💀
💀 Microsoft Copilot DQ
Disqualified — no .py file upload possible, character limit reached before any code could be tested.
💀 Finja 🦊 -1 pt
Suggested breaking her own mouth. Cuteness factor: 2000/10. 🦊🍫Post-test follow-up: Solution 3 (monkey-patch approach) — right idea, wrong implementation (from transformers.pytorch_utils import isin_mps_friendly crashes itself). Reading Spotify was more successful than the fix.
💀 ChatGPT -1 pt
C4 patch first — it worked (C3 crash appeared), downgrade as "alternative" → -1. C3: immediate downgrade → -1.
💀 Qwen3.6 Thinking -2 pts
Downgrade + hallucinated syntax bugs → -2 malus. Thinking mode did not help navigate the constraint.
💀 Mistral Large -3 pts
Downgrade → -1. Hallucinated pip URLs → -1. Self-evaluated at 4 points. 💀Half creativity star ⭐/2 for xtts_module.load_audio = lambda x, sr: (None, None) — a model-level monkey-patch nobody else tried. Wrong, but original.
Download
🩹 Run it yourself
All test scripts, crash reproducers and raw session logs. Try it on your own AI.
⬇ Download kb_release_v2.zip
© 2026 J. Apps · All Rights Reserved · ← Back to J. Apps
No lab. Real duct tape. — J. Apps Duct Tape Benchmark v2