Editor’s Note: This is the first post in a new series where we take a more in-depth and technical look at ways to improve the virtual reality experience. Today, we're deep diving into Equi-Angular Cubemaps, which is a new technique to stream higher quality VR video.

Since ancient times, cartographers have strived to make maps that accurately reflect the world. A central challenge of the art is projecting the Earth, which is curved, onto flat paper and screens. Many methods of projection have been proposed over the centuries, but to date no universally good answer has been found: every choice is always a tradeoff tuned for its intended use. Because new services like Google Maps and applications like VR video seek to provide accurate and meaningful information about our curved, 3D world on a 2D screen, this challenge persists even today.

While traditional cartography techniques must address the basic challenge of how to represent the world in 2D, video streaming also faces a new constraint that early mapmakers didn’t have: making efficient use of bandwidth. Streaming high quality video pushes the limits of network bandwidth, particularly for mobile networks. And when considering VR video, the bandwidth demands are vastly increased since it must represent imagery from a full sphere rather than just a small window onto the world. Stereo video roughly doubles the data yet again. Therefore making the most of available bandwidth is a top concern.

Equirectangular Projection

The equirectangular projection has the advantages of being both rectangular and straightforward to visualize. It’s also relatively easy to manipulate using existing video editing tools. However, when used for video transmission, it has serious problems. First, the poles get a lot of pixels, and the equator gets relatively few. This is challenging, because spherical videos usually have their important content distributed around the equatorial regions (the middle), which is the viewer’s horizon. It also has high distortion, which makes existing video compression technology work harder.

Traditional Cube Maps

The most straightforward way to do this is with a simple radial projection: you embed the sphere in a cube and project the image on the sphere outwards onto the surface of the cube.

In the figure above, the rays have all been equally spaced over the circle, but the points where the rays intersect the square are not equally spaced. As a result, the corners get more video pixels than the centers, because the longer blue line spans more pixels on the square edge than the red line, which again shortchanges the equator. In the full three dimensional case, the effect is even more pronounced.

The Equi-Angular Cubemap (EAC)

The rays emanating from the center represent a viewer’s line of sight, equally spaced by angular change. On the left is the traditional cubemap, where the mapping from face location to pixel location is linear. On the right is the more efficient EAC mapping.

The traditional cubemap has samples of varying length depending on the sample’s location on the cube face. EAC is specifically constructed to keep these lengths equal, creating uniformly allocated pixels. For all the reasons that cartography is hard, this 2D picture does not extend perfectly into 3D: if you choose to preserve one important feature of the mapping you invariably give up something else. Check out Conformal Mapping and Equal Area Projections for more information on preserving some characteristics at the cost of others. The EAC formula is mathematically precise in 2D, but only an approximation of an equal angle pixel distribution in 3D, albeit a pretty good approximation with manageable distortion.

Uniformity Comparisons

A useful method to visually compare different projection types is to use saturation maps. A saturation map shows a color-coded ratio of video pixel to display pixel density. The color coding goes from red to orange, yellow, green and finally blue. Green indicates an optimal pixel density ratio that’s near 1:1. Red, orange and yellow indicate insufficient density (too few video pixels for the available display pixels), and blue indicates wasted resources (too many video pixels for the available display pixels). You can change the overall colors on a saturation map by increasing the resolution of the video. But for a saturation map with lots of variation, as you bring the least saturated areas to green, you also increase the area of the video where resources are being wasted. Therefore, the ideal projection has a saturation map that is uniform in color - because it can be uniformly green with sufficient resolution.

Unsurprisingly, in the Equirectangular projection the poles are blue (wasteful, too many video pixels) and the equator orange (poor quality, too few video pixels).

In comparison, the Standard Cubemap moves the optimal green regions from near the poles towards the equator. The wasteful blue regions at the poles are gone altogether. On the other hand, the cubemap is better at the corners of the cube as compared to the center of its faces. Moreover, the equatorial region has the most variation in saturation, and the centers of the cube faces are actually lower quality than the worst regions of the equirectangular projection.

Finally, the EAC projection’s saturation is significantly more uniform than either of the previous two, while further improving quality around the equator. This uniformity allows maximal use of available bandwidth, thereby permitting a choice of whichever resolution delivers optimal pixel density in a given circumstance.

The Proof is in the Pudding

The Devil is in the Details...

An analogous operation is done for each face.

Now all six sets of [0, 1] texture coordinates need to be packed into a single video texture. If video shape and size are constrained by hardware or streaming limitations, these layouts can be nontrivial, and they may require analysis in their own right.  There are many different ways to layout the six faces of the cube into a video rectangle.  In addition, there might be discontinuities where faces meet each other, which can cause problems with video encoding and pixel interpolation when rendering. Texture atlases used in games have similar issues, and they are usually addressed by adding padding at the discontinuities. These details offer an opportunity for making choices and tradeoffs for the way the abstract EAC concept is deployed.

For those of you who are familiar with OpenGL, you may have noticed that this can all be done with a cubemap texture and a fragment shader which alters the texture lookups. You’re correct! The actual math is left as an exercise for the reader.

Editor’s Note: This is the first post in a new series where we take a more in-depth and technical look at ways to improve the virtual reality experience. Today, we're deep diving into Equi-Angular Cubemaps, which is a new technique to stream higher quality VR video.

Since ancient times, cartographers have strived to make maps that accurately reflect the world. A central challenge of the art is projecting the Earth, which is curved, onto flat paper and screens. Many methods of projection have been proposed over the centuries, but to date no universally good answer has been found: every choice is always a tradeoff tuned for its intended use. Because new services like Google Maps and applications like VR video seek to provide accurate and meaningful information about our curved, 3D world on a 2D screen, this challenge persists even today.

While traditional cartography techniques must address the basic challenge of how to represent the world in 2D, video streaming also faces a new constraint that early mapmakers didn’t have: making efficient use of bandwidth. Streaming high quality video pushes the limits of network bandwidth, particularly for mobile networks. And when considering VR video, the bandwidth demands are vastly increased since it must represent imagery from a full sphere rather than just a small window onto the world. Stereo video roughly doubles the data yet again. Therefore making the most of available bandwidth is a top concern.

Equirectangular Projection

The equirectangular projection has the advantages of being both rectangular and straightforward to visualize. It’s also relatively easy to manipulate using existing video editing tools. However, when used for video transmission, it has serious problems. First, the poles get a lot of pixels, and the equator gets relatively few. This is challenging, because spherical videos usually have their important content distributed around the equatorial regions (the middle), which is the viewer’s horizon. It also has high distortion, which makes existing video compression technology work harder.

Traditional Cube Maps

The most straightforward way to do this is with a simple radial projection: you embed the sphere in a cube and project the image on the sphere outwards onto the surface of the cube.

In the figure above, the rays have all been equally spaced over the circle, but the points where the rays intersect the square are not equally spaced. As a result, the corners get more video pixels than the centers, because the longer blue line spans more pixels on the square edge than the red line, which again shortchanges the equator. In the full three dimensional case, the effect is even more pronounced.

The Equi-Angular Cubemap (EAC)

The rays emanating from the center represent a viewer’s line of sight, equally spaced by angular change. On the left is the traditional cubemap, where the mapping from face location to pixel location is linear. On the right is the more efficient EAC mapping.

The traditional cubemap has samples of varying length depending on the sample’s location on the cube face. EAC is specifically constructed to keep these lengths equal, creating uniformly allocated pixels. For all the reasons that cartography is hard, this 2D picture does not extend perfectly into 3D: if you choose to preserve one important feature of the mapping you invariably give up something else. Check out Conformal Mapping and Equal Area Projections for more information on preserving some characteristics at the cost of others. The EAC formula is mathematically precise in 2D, but only an approximation of an equal angle pixel distribution in 3D, albeit a pretty good approximation with manageable distortion.

Uniformity Comparisons

A useful method to visually compare different projection types is to use saturation maps. A saturation map shows a color-coded ratio of video pixel to display pixel density. The color coding goes from red to orange, yellow, green and finally blue. Green indicates an optimal pixel density ratio that’s near 1:1. Red, orange and yellow indicate insufficient density (too few video pixels for the available display pixels), and blue indicates wasted resources (too many video pixels for the available display pixels). You can change the overall colors on a saturation map by increasing the resolution of the video. But for a saturation map with lots of variation, as you bring the least saturated areas to green, you also increase the area of the video where resources are being wasted. Therefore, the ideal projection has a saturation map that is uniform in color - because it can be uniformly green with sufficient resolution.

Unsurprisingly, in the Equirectangular projection the poles are blue (wasteful, too many video pixels) and the equator orange (poor quality, too few video pixels).

In comparison, the Standard Cubemap moves the optimal green regions from near the poles towards the equator. The wasteful blue regions at the poles are gone altogether. On the other hand, the cubemap is better at the corners of the cube as compared to the center of its faces. Moreover, the equatorial region has the most variation in saturation, and the centers of the cube faces are actually lower quality than the worst regions of the equirectangular projection.

Finally, the EAC projection’s saturation is significantly more uniform than either of the previous two, while further improving quality around the equator. This uniformity allows maximal use of available bandwidth, thereby permitting a choice of whichever resolution delivers optimal pixel density in a given circumstance.

The Proof is in the Pudding

The Devil is in the Details...

An analogous operation is done for each face.

Now all six sets of [0, 1] texture coordinates need to be packed into a single video texture. If video shape and size are constrained by hardware or streaming limitations, these layouts can be nontrivial, and they may require analysis in their own right.  There are many different ways to layout the six faces of the cube into a video rectangle.  In addition, there might be discontinuities where faces meet each other, which can cause problems with video encoding and pixel interpolation when rendering. Texture atlases used in games have similar issues, and they are usually addressed by adding padding at the discontinuities. These details offer an opportunity for making choices and tradeoffs for the way the abstract EAC concept is deployed.

For those of you who are familiar with OpenGL, you may have noticed that this can all be done with a cubemap texture and a fragment shader which alters the texture lookups. You’re correct! The actual math is left as an exercise for the reader.