Microsoft's Phi-2 2.7B model outperforms the recently announced Google Gemini Nano-2 3.2B model

Reading time icon 2 min. read


Readers help support MSPoweruser. When you make a purchase using links on our site, we may earn an affiliate commission. Tooltip Icon

Read the affiliate disclosure page to find out how can you help MSPoweruser effortlessly and without spending any money. Read more

Microsoft Phi-2

Over the past few months, Microsoft Research has been releasing a suite of small language models (SLMs) called “Phi”. The Phi-1 was released first with 1.3 billion parameters and it was specialized for basic Python coding. In September, Microsoft Research released Phi-1.5 model with 1.3 billion parameters, but it was trained with a new data source that included various NLP synthetic texts. Despite its small size, phi-1.5 was delivering a nearly state-of-the-art performance when comparable to other similarly sized models.

Today, Microsoft announced the release of Phi-2 model with 2.7 billion parameters. Microsoft Research claims that this new SLM delivers state-of-the-art performance among base language models with less than 13 billion parameters. On some complex benchmarks, Phi-2 matches or outperforms models up to 25x larger.

Last week, Google announced Gemini suite of language models. The Gemini Nano is Google’s most efficient model built for on-device tasks and it can run directly on mobile silicon. Gemini Nano-like small language model enables features such as text summarization, contextual smart replies, and advanced proofreading and grammar correction.

According to Microsoft, the new Phi-2 model matches or outperforms the new Google Gemini Nano-2, despite being smaller in size. You can find the benchmarks comparison between Google Gemini Nano-2 and Phi-2 models below.

ModelSizeBBHBoolQMBPPMMLU
Gemini Nano 23.2B42.479.327.255.8
Phi-22.7B59.383.359.156.7

In addition to outperforming Gemini Nano-2, Phi-2 also surpasses the performance of Mistral and Llama-2 models at 7B and 13B parameters on various benchmarks. Find the details below.

ModelSizeBBHCommonsense
Reasoning
Language
Understanding
MathCoding
Llama-27B40.062.256.716.521.0
13B47.865.061.934.225.4
70B66.569.267.664.138.3
Mistral7B57.266.463.746.439.4
Phi-22.7B59.268.862.061.153.7

While the previous two phi models were made available on Hugging Face, Phi2 has been made available on the Azure model catalog. You can learn more about Phi-2 here.

More about the topics: microsoft, microsoft research, Phi-1, Phi-2