OpenAI reveals breakthrough model

OpenAI has unveiled its newest model: The GPT-4o mini, which is its smallest AI model to date, costs less than its full-sized models, and performs better than GPT-3.5.

How does the GPT-4o mini perform?

GPT-4o mini scored 82% on an industry-recognized benchmark test, called the MMLU (Measuring Massive Multitask Language Understanding) which comprises 16,000 multiple-choice questions, across 57 academic subjects, and is designed to measure reasoning. This outperformed industry-leading small AI models, Gemini 1.5 Flash, which scored 79%, and  Claude 3 Haiku, which scored 75%. It even outperformed OpenAI’s flagship GPT-3.5, which only scored 70% on the test, but didn’t beat GPT-4o, which got 88.7% (to put this into context, Google claims that its Gemini Ultra scored 90%).

It is worth noting that some researchers take the MMLU benchmark test results with a pinch of salt because how the test is run differs from company to company, which doesn’t allow for a fair, like-for-like comparison. Plus, if the models being tested have been trained on the answers to the MMLU questions, there’s a strong chance they will cheat the system, and with no third-party evaluators involved in the process, the validity of the results can be questioned.

OpenAI states that its newest model is roughly the same size as Gemini 1.5 Flash and Claude 3 Haiku, but is much faster and cost-efficient.

It’s also said to be 60% cheaper and 2X faster to run than GPT-3.5 Turbo, making it “a compelling offering for speed-dependent use-cases including many consumer applications,” such as auto-suggestion features and data analysis tasks.

Why has OpenAI released a mini AI model?

Because compute costs to run the larger, more advanced AI models–like ChatGPT-4 Omni or Claude 3.5 Sonnet, for example–are astronomically high, developers are turning towards smaller models, that are often faster and more cost-efficient, while still capable of performing high-volume, simple tasks. OpenAI understands this and has released GPT-4o mini to offer developers something lightweight and inexpensive, making AI more accessible, which aligns with its broader mission:

“GPT-4o Mini really gets at the OpenAI mission of making AI more broadly accessible to people. If we want AI to benefit every corner of the world, every industry, every application, we have to make AI much more affordable.”