In-Air gesturing is likely to be one of the technology features coming to various devices in 2019 from smartphones to Mixed Reality headsets. Patently Apple posted a report on Friday Dec. 28th titled "Apple could be onboard the 2019 3D Camera trend for Smartphones that will introduce In-Air Gesture Controls." The report covered Sony's new 3D camera module that will be adopted in 2019 by smartphone OEMs starting with Huawei. Other OEMs are likely to follow and Apple was mentioned as reviewing the camera. One of the key features of this new 3D camera will be in-air gesturing as noted in photo below. A new Samsung patent application filed in Q4 2018 shows us that Samsung has another idea for in-air gesturing that involves using an advanced depth camera into a future VR headset.
Technology leaders such as Samsung, Apple (01, 02), Microsoft and Facebook have all shown an interest in developing in-air gesturing applications and devices like headsets that could work with such a feature.
Last month the U.S. Patent Office published a patent application covering gesture sensing in use with a mixed reality headset.
According to Samsung, various embodiments of the present invention cover a gesture sensing method of an electronic device that may comprise the operations of detecting an object, identifying a movement of the object and determining a gesture type corresponding to the movement of the object, activating a sensor for detecting a gesture corresponding to the gesture type according to the determined gesture type, and detecting the gesture through the activated sensor.
Further, an electronic device according to one of various embodiments of the present invention may be configured with a sensor unit for detecting an object and a movement of the object through at least one sensor; a camera module for photographing the object and the movement of the object through the at least one sensor; and a control unit configured to detect the object, identify the movement of the object, determine the gesture type corresponding to the movement of the object, activate the sensor for detecting the gesture corresponding to the gesture type according to the determined gesture type, and detect the gesture through the activated sensor.
An electronic device could identify a changing gesture intention in operations and activate only the sensors necessary for detecting an intended gesture or reset the sensors to detect the intended gesture. The in-air gesture system to control the system will also save on power usage.
Extended Mode
In Samsung's patent FIG. 4b presented above we're able to see a VR headset capable of detecting a shape of a hand as an object, and switch an operating mode from a default mode to an extended mode according to the detection of an object.
A headset operating in the extended mode can identify that at least one image of a hand shape received from at least a color and depth camera that corresponds to an image of a pre-stored simple gesture. Namely, the headset is able to identify that the movement of the object is intended as a simple gesture type.
The headset is able to detect a simple gesture #401 (in patent FIG. 4B above) in the extended mode without switching the mode. The headset operating in the extended mode can detect the simple gesture from a user and perform an operation according to the detected simple gesture.
As an embodiment, the simple gesture includes a pointing gesture and a grab gesture. The grab gesture may require a higher frame rate than the pointing gesture. Accordingly, even though the simple gesture is detected in the identical extended mode, it may be more effective to detect the grab gesture by setting a higher frame rate than the pointing gesture. Accordingly, the headset may have various sensor setting values in order to detect various simple gestures.
Full Performance Mode
Samsung then notes that in order to detect a user's complex gesture , the full performance mode may be a mode of setting the depth sensor with the highest frame rate value and the IR sensor to emit an IR light with the highest illumination value. The headset performing the full performance mode as illustrated in patent FIG. 4C below can detect the complex gesture #402, and perform an operation corresponding to the detected complex gesture.
Regarding FIG. 4C Samsung further describes that in order to detect a user's complex gesture the full performance mode may be a mode of setting the depth sensor with the highest frame rate value and the IR sensor to emit an IR light with the highest illumination value. The headset performing in the full performance mode is able to detect the complex gesture and perform an operation corresponding to the detected complex gesture.
Samsung's patent FIG. 5 below is a flowchart according to various embodiments of the present invention.
Samsung filed this patent application in Q2 2018. It was published by USPTO earlier in Q4 2018. One of the inventors listed on Samsung's patent filing is Ji Soo Yi, a VP at Samsung leading an AI Strategy Group that worked on Bixby 2.0 and other projects.
Comments