Meta's Llama 3.1 scores impressive head-to-head comparison to GPT-4o & Claude 3.5 Sonnet

Meta has just launched the Llama 3.1 series earlier this month, with both 405B and 70B variants. For an open-source model, that’s pretty impressive.

Large Model Systems Organization (LMSys Org), a crowdsourced open platform that compares large language models (LLMs), has extensively tested the two variants over the past week. And, as expected, both variants recorded amazing results.

Meta-Llama-3.1-405b-Instruct is ranked the third best on the Overall leaderboard alongside Gemini-Advanced-0514, making it the first open-source model to achieve such a milestone. The other two, GPT-4o and its mini variant remain the first-ranked, followed by Claude 3.5 Sonnet in the second place.

The 405B model registered impressive results in categories like coding, math, instruction-following, and hard prompts. In the English-only arena, as the open platform describes, Meta’s model even matches OpenAI’s GPT-4o. It’s the only model up there with an impressive 88.6 score on the MMLU benchmark, close to GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro.

Meta-Llama-3.1-70B-Instruct, surprisingly, also punches well above its weight. The model has been improving a lot since launch, ranked ninth alongside Claude 3 Opus and GPT-4.

LMSys Org tests the models by allowing the community to evaluate them through live, open, and community-driven pairwise comparisons. Created by LMSYS and UC Berkeley SkyLab in May 2023, the open-source platform has involved millions of users and tested over 90 chatbots.

Meta boss Mark Zuckerberg open-sourced its Llama models earlier this year. And now, the Facebook parent company launched the Llama 3.1-powered AI Studio, which lets you create your own AI based on your Instagram profiles.