Microsoft Research shows off an innovative pen-first UI for tablets

Satya Nadella recently claimed that Windows 10 X as embodied on the Surface Neo had a pen-first UI which he really enjoyed using.  From the little we have seen and what has been leaked regarding Windows 10 X however there appears to be little innovation when it came to the pen-based features of the operating system, with the main application being taking notes in OneNote on one screen while viewing and interacting with content on another.

This is a pity as Microsoft Research has of course been working for many years on very interesting new Natural User Interfaces combining Pen and Touch in very powerful ways.  One such recent demonstration is a way to add powerful sensing capabilities to tablets which let the operating system adapt to how users and holding the device, to adapt naturally to their intention.

In the demo, Microsoft Research proposes sensing techniques that transition between various nuances of mobile and stationary use via postural awareness. These postural nuances include shifting hand grips, varying screen angle and orientation, planting the palm while writing or sketching, and detecting what direction the hands approach from.

To achieve this, their system combines three sensing modalities:

1. raw capacitance touchscreen images,
2. inertial motion, and
3. electric field sensors around the screen bezel for grasp and hand proximity detection.

It includes interesting pen-first UI elements such as using the stylus as physical widget for “storing” and retrieving pictures and other clipboard items and a floating menu which is always present at the tip of your pen, no matter where you are on the screen.

See the demo below:

The research was published in the ACM CHI 2019 Conference on Human Factors in Computing Systems—where the work received an Honorable Mention award.

Do our readers agree that Windows 10 X and the Surface Neo is an opportunity for Microsoft to actually put some of their bold NUI ideas into practice? Let us know below.