Apple just submitted an application to the U.S. Patent and Trademark Office for a light field panoramic camera system for next-generation iPhones, iPads and head-mounted displays (HMD). With the camera, the user will be able to capture images (as well as position and orientation information) by holding up the device. Processing the images for relative positions and depth information, the light field panorama will allow a viewer to explore 3D views of the image with 6 degrees of freedom (DOF).

Patently Apple reports that, “this will be a killer app for iPhone cameras and future HMD.” “Using a rendering and viewing system such as a mobile device or head-mounted display, the viewer may see behind or over objects in the scene, zoom in or out on the scene, or view different parts of the scene,” it adds, noting that “Apple states that … a light field camera captures color intensities of light in a scene, and also the direction that the light rays are traveling in space.”

Google, which purchased light field photography/VR company Lytro in 2018, also plans on delivering a similar system in the future.

Light field photography differs from a conventional camera because the latter “records only light color intensities.” Among the types of light field cameras, one “uses an array of micro-lenses placed in front of an image sensor” and another relies on “multi-camera arrays.” Apple’s light field camera “provides various embodiments of methods and apparatus for capturing, processing, and rendering light field panoramas.”

Patently Apple notes that, in June 2017, it covered a story on “Apple Wins Key Patent for Light Field Cameras that could deliver an Immersive AR Experience for Macs, iDevices and Headset,” indicating the company, which filed the patent in 2016, has been working on this project for at least three-and-a-half years.