Qualcomm's Snapdragon Neural Processing Engine will enable better AR experiences

Apple ARKit is going to be largest AR platform in the world when it gets launched later this year. Hundreds of millions of iPhone and iPad devices will be compatible with ARKit, thanks to the powerful CPU and GPU that are already available on these devices. To enable highly accurate scene detection and object tracking, you need to run on-device neural network-driven algorithms. These algorithms cannot be made run entirely on CPU just like any other algorithm since they require more power and consume more battery. Since Apple exactly knows the hardware inside these devices, they have optimised the ARKit to run well on the iOS devices in a power efficient way.

Google on the other hand cannot do the same since their Android ecosystem is built on wide variety of CPU and GPU models from different vendors like Qualcomm, Samsung, MediaTek, etc, and their capabilities varies dramatically. Qualcomm, the most popular mobile processor maker is trying to solve this problem using the Neural Processing Engine (NPE) SDK. The Snapdragon NPE is a deep learning software framework designed for Snapdragon Mobile Platforms. It helps developers run one or more neural network models trained in Caffe/Caffe2 or TensorFlow on Snapdragon mobile platforms, whether that is the CPU, GPU or DSP. It supports both convolutional neural networks and custom layers.

• The Snapdragon NPE is engineered to help developers save time and effort in optimizing performance of trained neural networks on devices with Snapdragon.
• It does this by providing tools for model conversion and execution as well as APIs for targeting the core with the power and performance profile to match the desired user experience.
• Developers can take advantage of deep learning user experiences like style transfers and filters (augmented reality), scene detection, facial recognition, natural language understanding, object tracking and avoidance, gesturing, and text recognition to name a few.

Facebook for example is planning to integrate the Snapdragon NPE into the camera of the Facebook app to accelerate Caffe2-powered AR features. As a result, they can achieve 5x better performance on the Adreno GPU, compared to a generic CPU implementation resulting in a more fluid, seamless and realistic application of AR features when capturing photos and live videos.

Snapdragon NPE SDK includes the following:

• Android and Linux runtimes for neural network model execution
• Acceleration support for Qualcomm Hexagon DSPs, Qualcomm Adreno GPUs and Qualcomm Kryo, CPUs
• Support for models in Caffe, Caffe2 and TensorFlow formats³
• APIs for controlling loading, execution and scheduling on the runtimes
• Desktop tools for model conversion
• Performance benchmark for bottleneck identification
• Sample code and tutorials
• HTML Documentation

The Snapdragon NPE is compatible with Snapdragon 600 and 800 Series Mobile Platforms. According to my estimate, there will be at least 200 million active mobile devices running Snapdragon 600 and 800 series processors. Hopefully, developers will take advantage of this NPE platform to create great AR experiences on mobile devices. You can download the SDK here.

On a related note, Microsoft earlier this week announced that the next version of HoloLens will feature the second version of the HPU that will include an AI coprocessor to natively and flexibly implement Deep Neural Networks. Read more about it here.