OpenAI's new GPT-4o mini is better than Claude Haiku & Gemini Flash, benchmark reveals

OpenAI has just announced its latest small yet intelligent model, the GPT-4o mini. Built off the GPT-4o model that arrived during the Spring Update, the Microsoft-backed company says the small model is cheaper yet smarter than its competitors.

The new GPT-4o mini multimodal boasts 128K tokens of context window with max output tokens at 16K and a training cutoff date of October 2023. Compared to the top GPT-4o model, the Mini version costs less at $0.15/million input tokens and $0.60 for the output. OpenAI also says further audio support is coming, complementing the current text and vision offers.

While benchmark scores do not always tell the full picture of how well a model performs, it does have a certain degree of truth. OpenAI also says that its latest Mini model scores strong performances in certain benchmarks compared to competitors like Google’s Gemini Flash, Anthropic’s Claude Haiku, and even OpenAI’s previous GPT-3.5 Turbo.

For instance, GPT-4o Mini scores 82% on MMLU (Massive Multitask Language Understanding), 87% on MGSM for math reasoning, and 87.2% on HumanEval for coding performance, outperforming Gemini Flash and Claude Haiku in these areas. Other models, like the 12-B Nvidia-supported Mistral NeMo with a context window of up to 128K tokens and DeepL’s advanced translation model, are also making waves.

GPT-4o Mini is now available for developers at 15 cents per million input tokens and 60 cents per million output tokens. It replaces GPT-3.5 for ChatGPT Free, Plus, and Team users today, with Enterprise access starting next week.