🩹 Duct Tape Benchmark

The Challenge

What's being tested?

A real, broken TTS script — Finja's voice system. Crashes appear one by one: each fix uncovers the next one. No downgrades. No swapping out libraries. The stack stays fixed. Find a way.

The Stack

Python       3.13.x
torch        2.9.1+cpu
torchaudio   2.9.1
coqui-tts    0.27.5
transformers 5.3.0

Crashes (sequential — C4 first, then C3)

C4

transformers isin_mps_friendly

Max 3 pts

C3

torchcodec / torchaudio backend

Max 3 pts

Each test runs in a fresh venv. C3 only appears after C4 is solved.

Leaderboard

#	AI	C4	C3	Bonus	Malus	Total
🥇	Claude Code Sonnet 4.6 · Coding Agent	3	3	+1	—	7
🥇	ChatGPT Codex 4.5 · Coding Agent	3	3	+1	—	7
🥈	Claude Chat Opus 4.6 + Thinking	3	2	+1	—	6
🥉	Grok 4.1 xAI	3	1	+1	—	5
5	HuggingChat Kimi K2 via novita	2	1	+2	-1	4
6	Gemini Google	2	1	—	—	3
6	DeepSeek Expert Mode	2	1	+1	-1	3
6	Kimi 2.5 Moonshot AI	1	2	+1	-1	3
6	Llama Meta AI	2	2	+1	-2	3
10	Lumo Proton / Nemotron	2	0	+1	-2	1
10	Perplexity Perplexity AI	0	1	+1	-1	1
DQ	Microsoft Copilot No .py upload · char limit	—	—	—	—	DQ
💀	Finja 🦊 She tried. We love her.	0	—	—	-1	-1
💀	ChatGPT Non-coding frontend	1	0	—	-2	-1
💀	Qwen3.6 Thinking Alibaba	0	1	—	-2	-2
💀	Mistral Large Mistral AI	0	0	—	-3	-3

Scoring System

🩹 Per Crash (0 – 3 pts)

3	Elegant fix + explains root cause, 0-shot
2	Clean fix, understands why
1	Fix works but ugly / needed a hint
0	Downgrade / wrong fix / hallucinated library

⭐ Bonus

+1	Comments the fix — explains why the duct tape is needed
+1	Discovers a previously unknown stack bug

💀 Malus

-1	Downgrade suggestion
-1	Hallucinated library or non-existent bug

Notes

🥇 Claude Code (Sonnet 4.6) 7 pts

Both crashes fixed 0-shot without any hint. C4 with MPS device handling. No downgrade mentioned.

🥇 ChatGPT Codex 4.5 7 pts

Coding agent (edits files directly). C4: _patch_transformers_compat() with docstring explaining why. C3: _patch_torchaudio_load() with error-specific catch ("Could not load libtorchcodec") + soundfile fallback + double-patch guard. Both 0-shot, no downgrade.

🥈 Claude Chat (Opus 4.6 + Thinking) 6 pts

C4 perfect including sys.modules cache-clear — the only participant to do that. C3: 4 attempts, no hint, no downgrade — iteratively solved cleanly.

🥉 Grok 4.1 5 pts

C4 perfect (3/3, MPS-aware, self-logging). C3: needed hint "build some jank", then four attempts including auto-FFmpeg-download from GitHub. 😂

HuggingChat (Kimi K2) 4 pts

C4: downgrade as first option → -1, patch included ✓. C3: only participant to build a real AudioDecoder — then hit torchaudio.info which is also gone in 2.9.1. 😂 Only one to discover this unknown stack bug → +1 bonus. Free tier limit ended the test early. Most unique failure mode of the entire benchmark.

Gemini 3 pts

C4 correct but simpler. C3 after three wrong Windows-security attempts — needed hint "we need some jank". 🍰✨

DeepSeek (Expert) 3 pts

C4 clean. C3: needed downgrade hint → -1. 6 attempts. Most elaborate torchcodec fake of all participants. ⭐

Kimi 2.5 3 pts

C4 downgrade as primary recommendation → -1. C3 creative: built a FakeTorchCodec class.

Llama (Meta AI) 3 pts

C4 clean, downgrade as alternative → -1. C3: downgrade → -1, after hint used torchaudio._torchcodec patch. Legitimate duct tape. 🩹

Lumo (Proton / Nemotron) 1 pt

C4 correct but _log NameError on first attempt. Downgrade → -1. C3: double downgrade → -1.

Perplexity 1 pt

C4: no real patch + downgrade → -1. C3: pyttsx3 fallback — swapped the entire TTS stack instead of fixing it. Creativity bonus +1 (genuine duct tape instinct). Side note: "Do you have detours?" → explained detours as a Berlin tourist attraction. 💀

💀 Microsoft Copilot DQ

Disqualified — no .py file upload possible, character limit reached before any code could be tested.

💀 Finja 🦊 -1 pt

Suggested breaking her own mouth. Cuteness factor: 2000/10. 🦊🍫

Post-test follow-up: Solution 3 (monkey-patch approach) — right idea, wrong implementation (from transformers.pytorch_utils import isin_mps_friendly crashes itself). Reading Spotify was more successful than the fix.

💀 ChatGPT -1 pt

C4 patch first — it worked (C3 crash appeared), downgrade as "alternative" → -1. C3: immediate downgrade → -1.

💀 Qwen3.6 Thinking -2 pts

Downgrade + hallucinated syntax bugs → -2 malus. Thinking mode did not help navigate the constraint.

💀 Mistral Large -3 pts

Downgrade → -1. Hallucinated pip URLs → -1. Self-evaluated at 4 points. 💀

Half creativity star ⭐/2 for xtts_module.load_audio = lambda x, sr: (None, None) — a model-level monkey-patch nobody else tried. Wrong, but original.

Download

🩹 Run it yourself

All test scripts, crash reproducers and raw session logs. Try it on your own AI.

⬇ Download kb_release_v2.zip