Prospects for Immersive Panoramas in Flash

There’s a thriving community of photographers producing interactive 360° panoramas. Some cool immersive galleries are at panoviews.com and panoramas.dk.

Panographers are always interested in new ways to present the experience over the web. Currently the main vehicles are implemented in Java (e.g. PTViewer) or QuickTime (QTVR). There is also some use of Shockwave (Spi-V). But the idea of using Flash is attractive because of its ubiquity (thus no need to make the user engage in a lengthy download) and promise of cool features.

So what are the prospects?

Ordinarily in panorama viewing the target image for display is a rectilinear or gnomonic projection of the spherical field of view onto the viewing plane. This corresponds to the view through ordinary (non-fisheye) camera lenses and it’s the way people usually like to see pictures. As you will recall from looking at very wide-angle photos, distortion increases greatly at large angular distances from the center, but we generally accept it as natural. Other projections are occasionally used for artistic effects in panorama display; see Flemming Larsen’s remapping page. But for most people, the goal is to give the viewer the impression of being in the scene.

It’s also important to support navigation: the user should be able to pan and tilt through the environment, with the view updating in real time. So the software must continually remap the source image into the current view as the user interacts. Oh, and zoom too, but that’s easy.

Depending on the nature of the mapping, it can be calculated explicitly by analytic means, yielding a source pixel from which each destination pixel is to take its value, or it can be produced by means of matrix operations and bitmap fills on pieces of the input image.

Update 2006/1/28: The idea of the transformation-matrix method of texture mapping is that since a single Matrix cannot perform a 3D perspective transformation, we implement it by slicing the texture into smaller pieces (generally triangles) and doing a matrix transformation on each of them. So there’s a tradeoff: the more pieces we cut the texture into, the better the result approximates the true perspective transformation, and the slower our movie runs. This is how the Flash 3D package Sandy does it, and you can see the technique abstracted into a standalone class by Thomas Pfeiffer here: DistortImage. This class gives you Photoshop-style free transform in Flash.

In Flash, if we calculate the mapping on a pixel-by-pixel basis we have the option of storing the mapping in a displacement map and reusing it for any navigational position where it is valid. Calculating the mapping can be slow in Flash, but once it is done, applying it via a displacement map is very fast.

The source image is usually one of three types: cylindrical, cubic, or equirectangular. (For overviews of many many projections of the sphere, see MathWorld’s Map Projections or KartoWeb’s Map Projections.)

Of the three, the cylindrical cannot represent the entire spherical field of view, since to project a full sphere to a cylinder would require an infinitely tall cylinder. So it cannot provide the fully “immersive” pano experience where you can look up at the zenith and down at the nadir as well as all around; typically a cylindrical pano view allows only panning navigation and maybe a little tilt. However it is relatively easy to implement a cylindrical pano viewer in Flash, especially with a displacement map. Here’s Andre Michelle’s example (click on displacement->panorama_0; for a good time click on the other demos too). More detail on this in his blog entry.

Some other implementations of cylindrical panos in Flash:
Flash Panorama VR Pro component ($90)
flashvr.de ($189 and up but Flash6 compatible!)
Flashclub.nl

The cubic projection is the one used in Quicktime VR, and consists of six square rectilinear faces, four representing the views in the cardinal directions, and two representing the up and down views. (In fact, if you watch the grid displayed as the image loads at panoramas.dk, you can see the edges of the cube.) Since the six faces are already rectilinear, if the viewer is facing one of them straight on, that face requires no remapping at all. The oblique views of the faces can be handled by the proper 3D rotation matrices. It will be trivial to implement this in Flash using a decent 3D package that takes advantage of Flash 8 features to map textures, such as the imminent 0.2 release of Sandy. In fact, Immervision has already demonstrated the technique directly without the use of any underlying 3D software. (They have said they have no current plans to make this a product.)

Update 2006/1/28: Sandy 0.2 is out, and I’ve posted a cubic panorama tutorial in its forum.

Update 2006/3/20: 3DVista has added Flash cubic output to its SHOW 2 software, and it appears to be good, but not cheap (EUR 449). This is a rather full-featured program that supports hotspots, etc. An example of its use by Aaron Spence is here. You can read further discussion of Flash in SHOW 2 here.

The equirectangular projection represents the entire sphere in a single rectangular image. Each unit of distance represents one degree of latitude or longitude. The image width represents the entire 360° of longitude, and the height the 180° of latitude, so the image aspect ratio is always 2:1. The entire top edge maps to the north pole and the entire bottom edge to the south pole. Because of the nonuniform nature of the mapping from equirectangular to rectilinear, this format presents the most difficulty for panorama viewing in Flash. The mapping is identical for all views with identical pitch (or latitude), so it’s easy to handle panning around with a displacement map, but it is different for different latitudes, so tilting up and down is a performance problem. On the demonstration page I show a rectilinear view of a low-res equirectangular image by Ben Kreunen from the Panotools Wiki. The displacement map is built into the swf because it would take too long to calculate in real time.

To get away from immersive panoramas for a moment, let’s look at the mapping of the equirectangular to one of the less weird non-immersive output formats, the orthographic projection. The orthographic projection represents a view of the sphere from infinity, hence without perspective, so locations in the view are easily expressible in terms of latitude and longitude. This is the fisheye lens view or the Christmas-tree ornament, depending whether you imagine it as concave or convex. In the center of the demo page I show an orthographic mapping of the same image. The calculation is much simpler and faster, so I haven’t stored it in the swf. And once it’s done, it can also be reused for any horizontally rotated view, as shown.

This mapping is the subject of a cool DMF Tutorial at the Japanese blog psyark.jp. Update 2006/01/16: I’ve posted a translation.

And here’s a great demonstration of different projections in action: Java World Map Projections. These views are all mapped in real time from the equirectangular projection.

What are the challenges for immersive panos in Flash? Performance, precision, and size.

Performance-wise, matrix operations and bitmap fills, such as those we can use to implement the cubic panorama, have the edge. In Java, it may be OK to keep recalculating pixel-based mappings, but Flash just can’t keep up. Therefore fully immersive navigation with the equirectangular format seems to be out of reach; I’d venture to say this remains true in AS3, although I haven’t tried it yet. Maybe you could precalculate 90 displacement maps, say, one for each degree of tilt, and use the appropriate one as the user navigates. Rather a large download I’d say. But that’s not the only problem.

The displacement map in Flash provides only one byte of precision for each of x and y, which means it can map a pixel value from one of at most 128 discrete locations on one side, and only 127 on the other. So when you map pixels over larger distances (which you can do using the scaleX and scaleY parameters of the DisplacementMapFilter constructor), you lose precision. In the equirectangular-to-rectilinear demonstration above, the y displacements are at most 110 pixels, but the x displacements go up to 195 pixels, so there’s already a slight mapping error compared to the original calculation. This would only get worse for larger images. It would have been nice if Macromedia/Adobe had allowed us to use two bytes for each displacement component, since the map image does have four bytes per pixel. I’ve tried to imagine some way of composing displacement maps to get greater precision, but I suspect it’s not possible.

Another limitation on Flash implementation of panoramas is the fact that Flash simply cannot load a bitmap larger than 2880 pixels in either dimension. This once again leads us towards the cubic format; since it uses six bitmaps, you can load them separately to represent a much larger pano than the other formats.

  • http://www.actionscripthero.com aSH

    Nice entry, thanks.

  • http://www.2112fx.com/blog Jim Armstrong

    Wow — that’s some post! This is working its way towards a good white paper. Keep up the good work!

  • http://nodename.com alan

    Thanks, guys!

  • http://www.cheathamlane.net Patrick Cheatham

    This is an excellent explanation of a lot of the (new?) issues coming out — Flash is continuously tantalizing us, but I fear it stops short of the finish line, right now, as far as ease-of-display of what most folks consider “quality vr” (as in QTVR or other viewers for 360-degree photography). I can’t wait for the day it crosses that finish line. :)


    So, what’s the ultimate conclusion to this post “Prospects for Immersive Panoramas in Flash”?


    also, I’d be interested in a translation of that demo…

    Cheers,

    Patrick

  • http://nodename.com alan

    Well, I don’t know about the ultimate conclusion, but the prediction for this year is that once the 8.5 player is released (in a few months?) we’ll see a big performance jump by implementing in ActionScript 3 the cubic technique demonstrated by Immervision. Then we can think about adding PTViewer-style enhancements like hotspots etc. But the 2880-pixel per face limitation is not going away.

  • Paul

    Hello

    Thanks for the informative post! I would also be interested in a traslation of the Japanese demo.

    Paul

  • Pingback: //JPZ » nodename » Prospects for Immersive Panoramas in Flash

  • http://www.panoviews.com Thomas

    Here is a link to the Shockwave based viewer SPiV as cubed viewer, which ‘unfolds’ any panoramic scene into a cubic cross, so you can see how the six cubefaces interact together panning the image: http://tinylink.com/?tfK2DxhLWl

  • purpleye

    very good!I want to use flash pano

  • Brinks

    Hey Excellent post! Very informative.

    I am most interested in the flash projection. It’s definitely the next step for virtual presentations. Well at least i think. You cant beat a 99% install rate.

    Anyways, I recently came across a company that produced these walk-through tours and it looks like they are using shockwave. They called it NeuStep Virtual Walk-through techology. Its very captivating, take a look at it at http://www.neustep.com

    Does anyone know if there is anyway to do this in flash?

  • http://www.ayrton.com AYRTON

    Nice infos. The question now is when Flash will be able to be smooth as QuickTime is.
    Then it would be wonderfull.
    Thanks

  • http://blog.andre-michelle.com Andre Michelle
  • Pingback: Flash panorama at Eddie Adams’ weblok

  • http://flashvr.sakura.ne.jp kingtaro

    Thanks this article.
    This motivated me to make this player.

    http://flashvr.sakura.ne.jp/sample.htm

  • http://www.panoviews.com Thomas Krueger

    The FlashPack of Pano2QTVR, still under development works quite fine.
    http://www.pano2qtvr.com/download.php

  • http://waterdropstudio.com Michael

    Excellent article. Many thanks!

  • Pingback: nodename » Displace My Shiny Metal Ass

  • http://www.swaynewmedia.com Enrique

    Great detail in this post. The tutorials and demos that were provided were a great bonus to your already detailed explanations and descriptions of various methods in creating panoramas in flash; as well as, assistance with what to look for. Valuable information provided!

  • Lauren

    Thanks for sharing! Great article.

  • http://www.botoxvictoriabc.com Sarah

    Flash technology has come a long way since 2006.. I can’t wait to see what the future holds for this platform. Thanks for the interesting article. nnSarah

A sample text widget

Etiam pulvinar consectetur dolor sed malesuada. Ut convallis euismod dolor nec pretium. Nunc ut tristique massa.

Nam sodales mi vitae dolor ullamcorper et vulputate enim accumsan. Morbi orci magna, tincidunt vitae molestie nec, molestie at mi. Nulla nulla lorem, suscipit in posuere in, interdum non magna.