Meta's Llama 3.1 scores impressive head-to-head comparison to GPT-4o & Claude 3.5 Sonnet
The model is now available on Amazon Bedrock & Azure AI
2 min. read
Published on
Share this article
Improve this guide
Read our disclosure page to find out how can you help MSPoweruser sustain the editorial team Read more
Key notes
- Meta launched the Llama 3.1 series with 405B and 70B variants.
- LMSys Org ranked the 405B variant third-best overall, the highest for an open-source model.
- Meta also launched Llama 3.10-powered AI Studio, allowing users to create AI based on Instagram profiles.
Meta has just launched the Llama 3.1 series earlier this month, with both 405B and 70B variants. For an open-source model, that’s pretty impressive.
Large Model Systems Organization (LMSys Org), a crowdsourced open platform that compares large language models (LLMs), has extensively tested the two variants over the past week. And, as expected, both variants recorded amazing results.
Meta-Llama-3.1-405b-Instruct is ranked the third best on the Overall leaderboard alongside Gemini-Advanced-0514, making it the first open-source model to achieve such a milestone. The other two, GPT-4o and its mini variant remain the first-ranked, followed by Claude 3.5 Sonnet in the second place.
The 405B model registered impressive results in categories like coding, math, instruction-following, and hard prompts. In the English-only arena, as the open platform describes, Meta’s model even matches OpenAI’s GPT-4o. It’s the only model up there with an impressive 88.6 score on the MMLU benchmark, close to GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro.
Meta-Llama-3.1-70B-Instruct, surprisingly, also punches well above its weight. The model has been improving a lot since launch, ranked ninth alongside Claude 3 Opus and GPT-4.
LMSys Org tests the models by allowing the community to evaluate them through live, open, and community-driven pairwise comparisons. Created by LMSYS and UC Berkeley SkyLab in May 2023, the open-source platform has involved millions of users and tested over 90 chatbots.
Meta boss Mark Zuckerberg open-sourced its Llama models earlier this year. And now, the Facebook parent company launched the Llama 3.1-powered AI Studio, which lets you create your own AI based on your Instagram profiles.
