February 9, 2015, Electronic Imaging Conference, San Francisco—Ian Bickerstaff from Sony Computer Entertainment highlighted their efforts to bring stereoscope to the Playstation generation and encourage the move 3-D. The company has been developing the underlying technologies in their Project Morpheus and has included many of the technologies into the latest console.
A part of the effort has been to incorporate virtual reality capabilities to create a greater sense of presence in games. the critical components include displays with wide-field optics and accurate head tracking, and at least binaural if not full 3-D audio. The system also has to coordinate with the inputs from the tracked peripherals. This system could also be part of a social screen, with links to TV, tablets, or other mobile devices.
The concept of VR has been around for quite a while. The first patents for VR systems were filed in 1957 and included an air circulator to provide a "wind in the face" effect. Unfortunately, the technologies available at the time were not capable of the necessary performance. Now, the availability of LCD screens and high-performance compute for image generation can be coupled with the sensors and other system functions to make the concept real.
One key is the optics. Generally, the point of view for any image is placed at the screen plane. The focal point changes with the screen and actual viewing angle and position. For comfortable viewing, the alignment in both vertical and horizontal planes is critical and must match the expected field of view in a binocular framework.
The basic technology is very mature. The first systems appeared in 1849 with some lenticular stereoscopes that used a half lens for each eye to provide proper convergence. The stereoscopes of the mid-1800's were the first modern consumer hardware-software systems. Over time, improvements acromatic lenses which provided some prism to help with card misalignment and parallax adjustment. By 1900, Zeiss had a stereoscope that included capabilities for full adjustment for all likely optical adjustments, but turned out to be very hard to use.
Holman and Bates moved in another direction and provided adjustments for focus only. By the 1950's, Viewmaster made a stereoscope that had no adjustments and was a commercial success. They found that interpupillary distance is not critical to viewing comfort, so an interocular adjustment to move the lenses to accommodate the eye spacing was unnecessary. The internal alignment is much more important than any user adjustments since users are very tolerant to small errors.
These early systems had limited field of view, so were not capable of providing an immersive VR experience. Changing the optics to increase the field of view tends to increase distortion, consider a fish-eye lens as the extreme. The availability of high-performance graphics compute power allows for pre-distorting the images to cancel the optical distortions.
Another issue with wide fields of view is that of effective eye rotation. People have high-resolution in the central area of vision and lower resolution at the edges. People don't have to turn their heads to change the area of central focus, so some form of eye tracking is necessary to adjust the scent in the images for highest resolution in the area of central focus. The addition of pantoscopic tilt—the displacement of an image off axis from straight ahead—allows for the user to wear eyeglasses with the system, and also addresses the fact that most people don't look straight ahead when looking at a screen.
The various technologies are now being incorporated in the PS 4. A head tracking system provides a full 360° field of view that is implemented through accelerometer, gyroscope, and the PS camera. The full 6-axis accelerometer and gyroscope update at a 1 kHz rate but are subject to drift and lag. The camera tracks a LED at a 60 Hz rate for high accuracy. Together, the sensors and optical tracking provide the system with head tracking and orientation to enable the immersive view.
The system can generate and render the images in real time, but can have issues with latency and pipelining. Predictive algorithms help to reduce the lag, but still overshoot when you stop turning. The head tracking allows the system to adjust for the last instance errors through reprojection.
Consumer VR needs very high compute capabilities for it to be quickly adopted. The compute is needed for optical pre-distortion, head tracking, image generation, latency correction, and adjustments for parallax shift. A VR display can have a wider range of parallax than traditional 3-D displays, but comfort is still critical. Convergence requires matched focal lengths as well as critical vertical and horizontal alignment for stereo viewing in VR.
The image capture is another challenge. a wider field of view leads to linear distortion and has low resolution at the center where high resolution is needed. A highly distorted lens like a fish-eye lens is one alternative where the known corrections can be mapped to the distortions. An alternative is to use multiple cameras, but the alignments of the cameras is extremely difficult. The cameras can be in the same plane by redirecting images to the cameras through mirrors.
The problem with multiple cameras is stitching the images together for a seamless view. For still images, the computers can align the images on a line-by-line basis, but still have a problem encoding the parallax. Capture should work to minimize parallax in the scene so the processor only has to apply horizontal image translation corrections.
A typical 3-D system has a fixed screen and uses dynamic calculations to adjust the image at the camera. In comparison, VR uses a fixed camera and adjusts image parameters to fit the display. For capture, one issue is where is the camera crew going to be if you have a full 360° field of view? How do you direct attention through image generation and address the viewer's capability to look anywhere? In some sense, we have to recreate technologies for a VR world.