Bing announce new AI-based improvements in image search

Microsoft’s Bing search engine has announced new AI-based improvements to their Bing search engine, particularly the image search section. The improvements enable multi-granularity matches, improved understanding of user queries, images and webpages, as well as the relationships between them.

A bit like Google, Microsoft now incorporated BERT/Transformer technology leveraging 1) pre-trained knowledge to better interpret text information – especially for above mentioned hard cases; 2) attention mechanism to embed the image and webpage with the awareness of each other, so that the embedded document is a good summarization of the salient areas of the image and the key points on the webpage.

Microsoft also extracts a select set of object attributes from both query and candidate documents, and to use these attributes for matching.

This allows Microsoft to match the query {elderly man swimming pictures} with similar attributes from the image content and its surrounding text. Now the query and document can be considered a “precise match” since they share the same attributes. This is useful for more precise searches where users are searching for items with multiple specific features e.g. {blonde man with a mustache}, {dance outfits for girls with a rose})

Microsoft also automatically generate metadata for documents, including images, which is then more easily matched with searches. One of the most useful types of metadata is called “BRQ” – Best Representative Query. BRQs are typically a good summarization of the main topics of the webpage and the major image content. The process of generating BRQs for Bing images also heavily relies on many modern deep learning techniques.

Microsoft says the techniques have had a real impact on the usefulness and accuracy of their search engine, illustrated by the response of Bing to the query {car seat for Chevy impala 96} over the years.

Microsoft says Bing Image Search now offers a deeper semantic understanding of user queries. Deep learning buffs can read all the details at Microsoft here.