Earlier this summer the U.S. Patent & Trademark Office published a patent application from Microsoft that reveals a 3D depth camera for a future Surface device that will be able to interpret in-air hand gestures to control a function of the device or to manipulate an on-screen radial menu. The 3D camera could also capture input from a stylus writing or drawing objects on a surface directly in front of the Surface device as noted in our cover graphic. Microsoft notes that the user will also be able to draw with their finger or use a coffee stirrer to input data, beyond using a professional smart pen. Microsoft further notes that the user will also be able to draw with their finger or even use a coffee stirrer if need be to input data, beyond using a professional smart pen.
Microsoft's Patent Background
Touchscreens and other displays are found, among other places, in the medical field and in heavy industry, as well as in automated teller machines (ATMs), and kiosks such as museum displays or room automation, where keyboard and mouse systems do not allow a suitably intuitive, rapid, or accurate interaction by the user with the display's content.
Generally, portable electronic devices have relatively small touchscreens, leading to challenges in content creation and modification, such as in word processing, drawing, drafting, and so on. Display elements displayed on a touchscreen manipulated by a user's finger, which may be relatively large compared to the display elements, for example, may be at least partially occluded by the finger. Also, such manipulation by touch may be of a relatively low resolution due, at least in part, to the smallness of the touchscreen.
3D Object Tracking to Augment Display Area
Microsoft's invention relates to user interaction with elements. A user may write or draw on a surface in front of a device with a camera using any object such as a finger, pen, or stylus. The surface may also be used to simulate a partial or full size keyboard. The use of a camera to sense the three-dimensional (3D) location or motion of the object may enable use of above-the-surface (e.g., hover) gestures, entry of directionality, and capture (scan) of real objects into a document being processed or stored by the electronic device. One or more objects may be used to manipulate elements displayed by the portable electronic device.
In Microsoft's patent FIG. 9 we're able to see a tablet or smartphone that has a camera capturing the user drawing elements on a surface with a stylus ( could also be a coffee stirrer, hairpin etc..) that could then be added to a document or email that is open on the system at the time.
Microsoft notes that "A camera #912, which may be disposed in display device #906, may have a field of view that includes at least a portion of surface #908 (e.g., such as writing surface #910), stylus# 902, and hand #904. The field of view may also include the 3D space above surface #908 so that orientations and/or locations of stylus #902 and hand #904 may be detected. Thus, images or video (e.g., multiple images) captured by the camera may include objects in the space above surface, such as a stylus, hand and writing surfaces.
In the example, motion, location, and/or orientation of the stylus and hand may be used to operate a radial menu #914 that's presented on the display. In some cases, the radial menu may comprise various drawing colors, textures, features, edit commands, etc.
The camera in sync with a microphone can detect touch events and sounds related to the users actions that are to be interpreted to be included in a document or email on the device. This could include capturing drawings or writing or gestures as noted in other patent figures 5, 6, 7, 8 and 10 below.
Microsoft further notes that in some examples, the system's processor may use image processing techniques to determine a gesture a user is making in front of the camera. In some examples, the processor of the system may detect a shape, orientation, and/or position of hand by analyzing an image, or depth data associated with the image, that includes the hand. The processor may compare the object in the image to shapes and gestures in a memory using image processing techniques. Finding a reasonable match (e.g., a close match or a match of image and shape parameters that are within predetermined threshold values) may lead to a determination by the processor of a gesture that corresponds to the shape, orientation, and/or location of the hand.
The processor may then map or translate the gesture into a command that may affect execution of code, which may change how elements (e.g., text, audio, drawings, and so on) in display are being displayed, for example. In some examples, such gestures may be used to control pan and zoom of elements in the display. In other examples, such gestures may be used to rotate or translate elements in display. In still other examples, such gestures may be used for any of a number of editing processes for elements in display.
In some examples, the shape, orientation, and/or location of the hand may change over time and the processor of the system may measure such changes to determine speed or velocity of the hand or portions thereof. In this way, for example, the processor may detect user actions such as a hand being swiped to the left or to the right. In still other examples, the system may detect user actions such as a hand's rotation.
Microsoft's inclusion of radial menus is supported by a 2013 patent filing that was covered in a Patently Mobile report revealing their new GUI element in detail. Patent Bolt was our original non-Apple patent blog that was changed to Patently Mobile years later. One of the patent figures from our 2013 report on radial menus is noted below.
Microsoft's patent application that was published earlier this summer by the U.S. Patent Office was originally filed in Q1 2016. Considering that this is a patent application, the timing of such a product to market is unknown at this time.
Patently Mobile presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. About Posting Comments: Patently Mobile reserves the right to post, dismiss or edit any comments. Those using abusive language or behavior will result in being blacklisted on Disqus.
Comments