Microsoft applies for patent on combining Pen and Touch on Mobile Devices

In May last year Microsoft user interface research lead Bill Buxton and colleagues showed off a concept of how combining pen and fingers can improve the usability of Windows tablets and enhance productivity.

The video shows the benefit of how simply using the thumb of the hand holding your tablet can still allow powerful improvements in accessing functions in apps ranging from web browsers, drawing apps or even Excel spreadsheets.

Now a patent applied for around the same time has been published by the USPTO.

In the patent Microsoft notes:

To improve the functionality of touch enabled devices for accepting bimanual input in situations where the device is gripped or supported by one of the user’s hands, as well as other situations, thumb+pen inputs are described herein. The thumb of an engaging hand is identified and controls are placed within its range of motion to enhance the functionality provided by the free hand (referred to as the pen input). For instance, the actions of the thumb can be used to determine how pen actions made using the other hand are interpreted. In other aspects, the pen can indicate an object through pointing, while the thumb indirectly manipulates one or more of its parameters through touch controls. Marking menus, spring-loaded modes, indirect input, and conventional multi-touch interfaces are applied with respect to the bimanual input mode in which one hand is positioned to hold or support the device, and the other hand is free.
The functionality of the user’s device is improved by the present disclosure in at least that an improved user experience is provided that enables the user to manipulate virtual objects displayed on the device with bimanual input while one hand is otherwise engaged in manipulating the physical object of the device. The range of inputs available to the user via the combined thumb of the engaged hand and the input device of (or held by) the free hand are greater than those available to either alone, and thus the efficiency at which the user can manipulate content is improved, and fewer computer resources are expended in effectuating user commands.

They note the thumb is available and sufficiently mobile to manipulate many controls, enabling a whole new space of “thumb + pen” interactions. This can allow one to readily interleave use of a pen between annotation and cell-selection on a spreadsheet, for example, or to select cells and copy them to another (possibly distant) location on the sheet—thus illustrating how pen and touch (via the thumb) can afford a far more casual form of productivity, even when one is just kicking back on the couch with such devices.

The Excel demo in the early part of the video is particularly interesting, as it seems to use adaptations of Microsoft’s Surface Dial radial control, which is of course already built into Windows and which is gaining increasing support from developers.

Microsoft published their research in the Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17) and the paper can be read here. The full patent can be read at the USPTO here.