Microsoft Research inspire video captioning bot
2 min. read
Published on
Share this article
Improve this guide
Read our disclosure page to find out how can you help MSPoweruser sustain the editorial team Read more
Two Chinese researchers, Chia-Wen Lin and Min Sun, professors in the Electrical Engineering department of National Tsinghua University in Taiwan, have created a bot which will watch your video for you, determine its highlights, create a relevant title for easy searching, and recommend who might want to be tagged to watch it.
Inspired by Microsoft Research’s COCO (Common Objects in Context), Professor Lin and Professor Sun collaborated with Dr. Tao Mei, a lead researcher in multimedia at Microsoft Research Asia in 2015, using COCO captions for sentence augmentation and using captions in MSCOCO to train their system.
Their own system analyzed 18,000 videos for highlights and generated 44,000 titles/descriptions. To improve the system, Professor Sun and his students participated in the VideoToText challenge sponsored by Microsoft Research, using the data released in the challenge for additional validation.
“Our research has taken us one step closer to the holy-grail of visual intelligence, understanding visual content in user-generated videos,” said Professor Sun.
Their research will eventually enable you to easily find and watch user-generated content on social media, including that amazing clip of your grandpa losing his teeth while dancing at your cousin’s wedding.
Professor Sun and Dr. Tao Mei’s research have been published at Arxiv.org here.
