Yesterday the U.S. Patent & Trademark Office published a patent application from Microsoft that revealed a very sophisticated, next generation home automation device that is driven by Cortana, Microsoft's digital assistant. It's a home automation system that leap frogs Amazon's Echo driven by Alexa and Google Home driven by Google Assistant. Way beyond voice activation, Microsoft's system will offer a highly sophisticated 3D depth camera in tune with a pico projector to allow video conferencing to occur in any room in your home or office as the imagery could be projected on a wall instead of a computer display as noted in our cover graphic. Microsoft's home automation system could be set up in a user's home or office and act as a form of video-intercom. The system could be set up in a home in various configurations such as a device like the Amazon Echo or on the wall or ceiling so that a conversation could take place while a user is moving from room to room.
Overview of Microsoft's Home Automation System +
Screen-based desktop videoconferencing, screen-based video presentations and screen-based digital television are now mature technologies, but the requirement for a screen is taxing in terms of space requirements, mobility, and cost. Using smart phones or tablet computers for mobile video conferencing allows for mobility but introduces the difficulty of holding a screen while moving, small image size and difficulty multi-tasking.
More recently, augmented reality has become available as a way of presenting video or other image content. However, augmented reality headsets can be restrictive in some situations and can be difficult to use by novice users, or those with physical disabilities, the elderly and the very young. In the case of augmented reality video conferencing, augmented reality headsets impede the view of the face of both participants if both use headsets, or one participant if that person is not wearing a headset.
Microsoft's remedy is found in one of their latest published inventions that relate to a portable projection unit which has a rotating capture module and a rotating projection module.
The capture module has at least one color camera, at least one microphone and at least one depth camera and is configured to capture images of an environment of the projection unit.
Microsoft's patent FIG. 2 as noted above is a schematic diagram of a room with a projection unit #100 on a table. The projection unit is seen projecting an image #204 onto a wall in the room. In this case the image is of a person #206 who is a remote party of a video call ongoing between the projection unit and another communications device such as a smart phone, laptop computer, meeting room video conferencing equipment, tablet computer or other video conferencing device.
The projection unit is capturing images of the seated user as indicated by triangle #202 and transmitting the captured images to the remote party. In this way screen-less video conferencing is enabled in a simple manner without the need for augmented reality headsets.
The size and shape of the projection unit and the ability to dynamically capture and dynamically project, enable the user to operate the projection unit whilst they carry on a natural conversation eye to eye. The user looks over the projection unit to see the images of the remote participant.
In Microsoft's patent FIG. 3 we're able to see a perspective view of the projection device (unit 100). The projection device is seen as cylindrical and comprises a plurality of cylindrical modules #300, #302, #304, #306, #308 in a stacked arrangement. It's overall appearance is something like Amazon's Echo.
One or more of the cylindrical modules are rotatable about the axis of the cylinder. In some examples, at least two of the cylindrical modules are independently rotatable with respect to one another, about the axis of the cylinder. The projection unit has a flat base which can rest on a table top and, in some examples, is sized and shaped to be portable. The cylindrical design of FIG. 3 is an example only; other designs of projection unit can be used.
The projection unit 100 also comprises a rotating projection module #304 comprising at least one pico projector and optionally comprising an optical arrangement comprising mirrors and/or lenses configured to optically move light projected by the pico projector.
In some examples the rotating projection module is rotating by virtue of the fact that the whole module rotates about its center. In some examples the rotating projection module has the property of rotating by virtue of the fact that the projected light it emits is rotatable with respect to the pico projector by using the optical arrangement. In some examples the rotating projection module rotates about its center and also has an optical arrangement for rotating the projected light with respect to the pico projector.
The projection device comprises one or more loudspeakers. In the example of FIG. 3 cylindrical component #302 is a 360.degree speaker section comprising one or more loudspeakers. In some examples the projection device has one or more loudspeakers where the direction of sound output can be controlled such that the sound appears to be originating from a specified location in the environment.
The projection device comprises a rotating capture module which may be housed in cylindrical component #300. The rotating capture module comprises at least one depth camera, at least one color camera, and at least one microphone.
Microsoft notes that the optical arrangement may be drivable to dynamically change the field of view of the capture module. For example, the field of view may be rotated to capture the gaze of a local user in such a manner that the local user appears to be looking at the projection unit, rather than over, under or beside the projection unit. This helps facilitate the perception of mutual direct gaze by both a local and a remote user.
The projection device comprises a communications component which may be part of the computing device #306 or may be provided in a cylindrical component #308 of the stacked arrangement of FIG. 3. The communications component is configured to enable the projection unit to communicate with other entities over a communications network using wired and/or wireless communications.
Overview of the Camera System
Microsoft's patent FIG. 4 is a schematic diagram of functional components of an example projection unit.
The scene reconstruction module #414 (purple box) comprises software and/or hardware which receives depth and/or color image data from the rotating capture module and computes characteristics of surfaces in the environment such as 3D position, size, texture, color, reflectivity, intensity, position with respect to other surfaces in the environment.
For example it may compute a 3D reconstruction of surfaces in the environment by combining data from frames of depth image data in sequences of frames captured by the depth camera. The 3D reconstruction may comprise a truncated signed distance function or other representation of a surface. The 3D reconstruction holds information about 3D position of surfaces in the environment, relative sizes of the surfaces, and relative positions of the surfaces with respect to one another.
The scene reconstruction module may compute an indication of reflectivity of surfaces in the environment from the depth and/or color images. For example, by assessing the amount of multi path reflection observed in depth images captured by time of flight depth cameras, by assessing the amount of light returned to the depth and/or color cameras under active illumination by a light source (such as the pico projector) at the projection unit in conditions when the ambient illumination in the environment is low, by assessing intensity levels of the color images with respect to calibration color images captured under specified conditions or by using high level data such as a floor plan of the building.
The scene reconstruction module may compute an indication of texture of surfaces in the environment from the 3D reconstruction by assessing smoothness of surfaces in the 3D reconstruction. It may also use the color images to compute texture of the surfaces.
The scene reconstruction module may compute color and intensity of surfaces in the environment from the color images.
Another key aspect of the camera system includes the gaze tracker #418 which may comprise body part or skeletal tracking technology such as increasingly used with depth cameras and/or color cameras. The gaze tracker detects the 3D pose of a user's head using the body part or skeletal tracking technology, where the 3D pose is a six degree of freedom parameter indicating 3D position and orientation. From the 3D head pose the gaze tracker computes gaze direction of the user and field of view of the user. In some examples the field of view takes into account scene reconstruction data, so that if a user's field of view is partially blocked by an object in the environment, the field of view is restricted.
Different Usage Scenarios
Microsoft's patent FIG. 5 noted above is a schematic diagram of a scenario with a laptop computer and a projection unit on a table, and of another scenario with a person walking towards a projection unit on a table; Patent FIG. 6 noted below is a schematic diagram of a scenario in which a user moves with respect to a projection unit, and of a scenario in which a user walks from one room into another room, both rooms having a projection unit.
Alternative Wall System
Microsoft's patent FIG. 1 noted below is a schematic diagram of two rooms in different buildings, each room having a projection unit which projects images of a video call onto a surface in the respective room. This system setup is different in that the camera systems are either wall or ceiling set instead of being a portable device.
A Note for Tech Sites covering our Report: We ask tech sites covering our report to kindly limit the use of our graphics to one image. Thanking you in advance for your cooperation.
Microsoft filed their patent application back in July 2015. Considering that this is a patent application, the timing of such a product to market is unknown at this time.
Patently Mobile presents a detailed summary of patent applications with associated graphics for journalistic news purposes as each such patent application is revealed by the U.S. Patent & Trade Office. Readers are cautioned that the full text of any patent application should be read in its entirety for full and accurate details. About Posting Comments: Patently Mobile reserves the right to post, dismiss or edit any comments. Those using abusive language or behavior will result in being blacklisted on Disqus.