Email Bench

Evaluates language models on producing production-ready emails from real briefs, in HTML and React Email.

0%10%20%30%40%50%60%$0.005$0.01$0.02$0.04$0.08$0.16$0.32$0.64$1.28Average cost per task (log scale)GPT-6 Astra: 50.5%, $0.64 average cost per taskGPT-6 Sol: 47.5%, $0.12 average cost per taskClaude Opus 5.5: 36.5%, $0.44 average cost per taskGPT-6 Luna: 36.0%, $0.006 average cost per taskGrok 4.7: 33.5%, $1.20 average cost per taskClaude Fable 5.1: 25.0%, $0.99 average cost per taskMuse Spark 1.3: 18.0%, $0.15 average cost per taskGemini 3.8 Flash: 15.5%, $0.82 average cost per taskDeepSeek V4.1 Flash: 13.0%, $0.02 average cost per taskGLM 5.3: 6.0%, $0.37 average cost per taskGLM 5.3 Flash: 4.0%, $0.02 average cost per taskGPT-6 AstraGPT-6 SolClaude Opus 5.5GPT-6 LunaGrok 4.7Claude Fable 5.1Muse Spark 1.3Gemini 3.8 FlashDeepSeek V4.1 FlashGLM 5.3GLM 5.3 Flash

200 tasks · 100 HTML · 100 React Email

Model
1GPT-6 Astra50.5%$0.64124,773
2GPT-6 Sol47.5%$0.12109,496
3Claude Opus 5.536.5%$0.44260,365
4GPT-6 Luna36.0%$0.006117,770
5Grok 4.733.5%$1.201,653,584
6Claude Fable 5.125.0%$0.99261,027
7Muse Spark 1.318.0%$0.15197,504
8Gemini 3.8 Flash15.5%$0.823,368,863
9DeepSeek V4.1 Flash13.0%$0.02733,785
10GLM 5.36.0%$0.37925,640
11GLM 5.3 Flash4.0%$0.02283,080

Every model runs each task once through the Pi coding agent, at high reasoning effort. A run that produces no email counts as a failure. Cost and tokens are averages per task.