It was reported last year that Google was acquiring Lytro. Then the story shifted to Google acquiring some Lytro patents, more specifically 59 light-field and imaging patents. Lytro had technology that mastered refocusing images after a photos was taken, allowing the user to shift the bokeh effect to anything or anyone in the photo that could dramatically alter the original shot. A simply example of refocusing photos is seen in our cover graphic above.
In the video below about Lytro, going to the 3:50 mark will show you how a user using a PC was able to simply click on any part of a photo using Lytro's software to change the focus of the photo.
This could be a feature that a future Pixel phone camera may be able to deliver.
Last month a Google patent application was published by the US Patent and Trademark Office (USPTO) titled "Capturing light-field Images with Uneven and/or Incomplete Angular Sampling." The patent may be interconnected with what was presented at the top of our report being that in their patent summary its noted that the invention will "increase the quality of the depth data, extend the refocusable range of the system."
In the background to the invention they note that light-field cameras may be used to capture a four-dimensional (4D) light-field image, with two spatial dimensions, x and y, and two angular dimensions, u and v. In many light-field cameras, a plenoptic microlens array and a single photosensor, containing a two-dimensional (2D) array of pixels, are used. These plenoptic light-field cameras capture image data that may be much more versatile than traditional two-dimensional image data.
In particular, a light-field image may be processed to create a set of virtual views, in which focus distance, center of perspective, depth-of-field, and/or other virtual-camera parameters are varied within ranges enabled by the data in the four-dimensional light-field image. Further, the light-field data may be analyzed to calculate a depth map and/or analysis information.
One drawback of existing plenoptic cameras is that virtual views have relatively low resolution. In order to capture the four-dimensional light-field, the spatial resolution is decreased significantly in order to increase the resolution of the angular dimensions. In many applications, the output resolution of virtual views may be too low for widespread adoption.
However, depending on the use case, a complete and even sampling in the two angular dimensions may not be required. In many cases, the desired output from the light-field camera may be a combination of high-resolution image data and depth data, often in the form of a depth map or a set of three-dimensional (3D) points (a point cloud). In some use cases, the desired output may be high-resolution depth data.
Google's Patent Summary
Various embodiments of the described system capture light-field images with uneven and/or incomplete angular sampling. Such embodiments may increase spatial and/or output resolution, increase the quality of the depth data, extend the refocusable range of the system, and/or increase the optical baseline. Some of the embodiments utilize novel exit pupil shapes and/or configurations of the microlens array.
Google's patent FIG. 1 below is a diagram of a plenoptic light-field camera; FIGS. 2A to 2E are diagrams of various aspects of the plenoptic light-field camera of FIG. 1.
All three engineers listed on this US patent were previous Lytro engineers/inventors as noted below:
1: Kurt Akeley "Distinguished Engineer" former Lytro CTO
2: Chia-Kai Liang, Senior Staff Software Engineer; former Architect, Computational Photography at Lytro
3: Colvin Pitts Senior Staff Software Engineer at Google; former 10 year Senior Architect as Lytro
Google acquired this 2018 granted patent from Lytro as noted in screenshot below from the USPTO.
While the base of this invention was granted to Lytro and transferred to Google in 2018 as noted above, the patent application that was published on April 25, 2019 is "technically" now a "continuation patent."
A continuation patent always means that the company is adding new aspects, methods, technologies, materials and so forth to the original invention. The base of the patent is protected by the granted patent.
In this recent continuation patent, Google "cancels" the original patent claims 1-29 and presents 17 new patent claims #30-47 as presented below.
- A light-field camera comprising: an aperture configured to receive incoming light and having a rectangular exit pupil; an image sensor; a microlens array disposed between the aperture and the image sensor, wherein the microlens array comprises a plurality of microlenses arranged in a plurality of rows, each row arranged at a non-zero acute angle relative to the rectangular exit pupil; and wherein the image sensor is configured to generate light-field data based on the incoming light received through the microlens array.
- The light-field camera of claim 30, wherein the plurality of microlenses comprises a plurality of rectangular microlenses.
- The light-field camera of claim 30, wherein the plurality of microlenses comprises a plurality of circular microlenses.
- The light-field camera of claim 30, wherein the plurality of microlenses comprises a plurality of hexagonal microlenses.
- The light-field camera of claim 30, wherein a width of the rectangular exit pupil is greater than a length of the rectangular exit pupil.
- The light-field camera of claim 30, wherein the rectangular exit pupil is shared and oriented, relative to the microlens array, such that the incoming light received through the microlens array forms a tessellated pattern at the image sensor.
- The light-field camera of claim 30, further comprising: a main lens through which the incoming light is to pass through prior to redirection by the microlens array.
- The light-field camera of claim 30, further comprising: a masking system configured to position at least one of a plurality of masks proximate the aperture, wherein the masks define a plurality of exit pupils having a plurality of different shapes, each of which defines an exit pupil for the aperture, the plurality of exit pupils including the rectangular exit pupil.
- A light-field camera comprising: an aperture configured to receive incoming light and having a rectangular exit pupil; an image sensor; a microlens array disposed between the aperture and the image sensor, wherein the microlens array is rotated by a non-zero acute angle relative to the rectangular exit pupil; and wherein the image sensor is configured to generate light-field data based on the incoming light received through the microlens array.
- The light-field camera of claim 38, wherein microlens array comprises one of: a plurality of rectangular microlenses; a plurality of circular microlenses; and a plurality of hexagonal microlenses.
- The light-field camera of claim 38, wherein a width of the rectangular exit pupil is greater than a length of the rectangular exit pupil.
- The light-field camera of claim 38, wherein the rectangular exit pupil is shared and oriented, relative to the microlens array, such that the incoming light received through the microlens array forms a tessellated pattern at the image sensor.
- The light-field camera of claim 38, further comprising: a main lens through which the incoming light is to pass through prior to redirection by the microlens array.
- The light-field camera of claim 38, further comprising: a masking system configured to position at least one of a plurality of masks proximate the aperture, wherein the masks define a plurality of exit pupils having a plurality of different shapes, each of which defines an exit pupil for the aperture, the plurality of exit pupils including the rectangular exit pupil.
- A computer-implemented method comprising: generating, at a processing system, a two-dimensional (2D) image from captured light-field data, the light-field data comprising a first spatial dimension, a second spatial dimension, a first angular dimension, and a second angular dimension, and wherein a first resolution of the light-field data in the first angular dimension is larger than a second resolution of the light-field data in the second angular dimension.
- The method of claim 44, further comprising: capturing the light-field data using a light-field camera having an aperture configured to receive incoming light and having a rectangular exit pupil, an image sensor, and a microlens array disposed between the aperture and the image sensor, wherein the microlens array comprises a plurality of microlenses arranged in a plurality of rows, each row arranged at a non-zero acute angle relative to the rectangular exit pupil.
- A non-transitory computer readable medium embodying a set of executable instructions to: generate, at a processing system, a two-dimensional (2D) image from captured light-field data, the light-field data comprising a first spatial dimension, a second spatial dimension, a first angular dimension, and a second angular dimension, and wherein a first resolution of the light-field data in the first angular dimension is larger than a second resolution of the light-field data in the second angular dimension.
- The non-transitory computer readable medium of claim 46, wherein the set of executable instructions further are to: capture the light-field data using a light-field camera having an aperture configured to receive incoming light and having a rectangular exit pupil, an image sensor, and a microlens array disposed between the aperture and the image sensor, wherein the microlens array comprises a plurality of microlenses arranged in a plurality of rows, each row arranged at a non-zero acute angle relative to the rectangular exit pupil.
For engineers or photographers that are tech-savvy, check out Google's (continuation) patent #20190124318 for more details and patent figures. The patent application that was published by USPTO on April 25, 2019, was originally filed on July 11, 2018.
Note: Our Cover graphic of refocusing a photo after it was taken is not a Lytro example. Once Lytro closed shop and their IP was acquired by Google, all major examples of Lytro's refocusing photos disappeared on Google search. The only example of refocusing a photo after it was taken that we could find from Lytro was in the video that we presented in our report.
Comments