Venturebeat has tested the language proficiency of the leading personal digital assistants, and has found a lot of room for improvement.
Looking at Siri, Cortana and Google Now, they tested it with 18 questions in English, Italian, French, German and Mandarin, giving full marks for a succinct answer, half points for a relevant and correct web search and no points for a complete failure.
All three scored relatively well in English, but things fell apart for Google and Microsoft when they moved overseas, while Apple managed to keep its performance up relatively good in all languages.
First up was English. Not surprisingly, English had the highest accuracy rates across all assistants. Siri scored on top (14.5/18) but both Google Now (11.5) and Cortana (12.5) weren’t trailing too far behind.
Next came Italian. Congratulations to Siri, who unequivocally left her competitors in the dust. Apple’s localization team should be proud because Siri scored identically to English (14.5/18). Google Now, on the other hand, scored a less than impressive 8.5/18 and our emerging star was suddenly not so remarkable anymore, scoring 6/18.
French and German results were similar to Italian. Siri dropped slightly in German and French and Google Now and Cortana improved marginally in both languages.
Last came Mandarin. We suspected that this might be the deciding factor that would set apart the boys from the men. Siri, again, scored on top with a 13.5 (only 1 point below its English score). Cortana scored a low 5, which given it’s performance in Italian, German & French, isn’t so surprising. Google, however, averaging a score of about 10/18 until now, dropped to a disconcerting score of 1.5/18.
They note that Cortana was very likely to offer a web search rather than an answer, which we suspect may have cut its score, but given that the expectation these days is for a direct answer rather than a search this is clearly an area where Microsoft needs to improve.
How does Cortana perform in your language? Let us know below.