Panographers are always interested in new ways to present the experience over the web. Currently the main vehicles are implemented in Java (e.g. PTViewer) or QuickTime (QTVR). There is also some use of Shockwave (Spi-V). But the idea of using Flash is attractive because of its ubiquity (thus no need to make the user engage in a lengthy download) and promise of cool features.
So what are the prospects?
Ordinarily in panorama viewing the target image for display is a rectilinear or gnomonic projection of the spherical field of view onto the viewing plane. This corresponds to the view through ordinary (non-fisheye) camera lenses and it’s the way people usually like to see pictures. As you will recall from looking at very wide-angle photos, distortion increases greatly at large angular distances from the center, but we generally accept it as natural. Other projections are occasionally used for artistic effects in panorama display; see Flemming Larsen’s remapping page. But for most people, the goal is to give the viewer the impression of being in the scene.
It’s also important to support navigation: the user should be able to pan and tilt through the environment, with the view updating in real time. So the software must continually remap the source image into the current view as the user interacts. Oh, and zoom too, but that’s easy.
Depending on the nature of the mapping, it can be calculated explicitly by analytic means, yielding a source pixel from which each destination pixel is to take its value, or it can be produced by means of matrix operations and bitmap fills on pieces of the input image.
Update 2006/1/28: The idea of the transformation-matrix method of texture mapping is that since a single Matrix cannot perform a 3D perspective transformation, we implement it by slicing the texture into smaller pieces (generally triangles) and doing a matrix transformation on each of them. So thereâ€™s a tradeoff: the more pieces we cut the texture into, the better the result approximates the true perspective transformation, and the slower our movie runs. This is how the Flash 3D package Sandy does it, and you can see the technique abstracted into a standalone class by Thomas Pfeiffer here: DistortImage. This class gives you Photoshop-style free transform in Flash.
In Flash, if we calculate the mapping on a pixel-by-pixel basis we have the option of storing the mapping in a displacement map and reusing it for any navigational position where it is valid. Calculating the mapping can be slow in Flash, but once it is done, applying it via a displacement map is very fast.
The source image is usually one of three types: cylindrical, cubic, or equirectangular. (For overviews of many many projections of the sphere, see MathWorld’s Map Projections or KartoWeb’s Map Projections.)
Of the three, the cylindrical cannot represent the entire spherical field of view, since to project a full sphere to a cylinder would require an infinitely tall cylinder. So it cannot provide the fully “immersive” pano experience where you can look up at the zenith and down at the nadir as well as all around; typically a cylindrical pano view allows only panning navigation and maybe a little tilt. However it is relatively easy to implement a cylindrical pano viewer in Flash, especially with a displacement map. Here’s Andre Michelle’s example (click on displacement->panorama_0; for a good time click on the other demos too). More detail on this in his blog entry.
The cubic projection is the one used in Quicktime VR, and consists of six square rectilinear faces, four representing the views in the cardinal directions, and two representing the up and down views. (In fact, if you watch the grid displayed as the image loads at panoramas.dk, you can see the edges of the cube.) Since the six faces are already rectilinear, if the viewer is facing one of them straight on, that face requires no remapping at all. The oblique views of the faces can be handled by the proper 3D rotation matrices. It will be trivial to implement this in Flash using a decent 3D package that takes advantage of Flash 8 features to map textures, such as the imminent 0.2 release of Sandy. In fact, Immervision has already demonstrated the technique directly without the use of any underlying 3D software. (They have said they have no current plans to make this a product.)
Update 2006/3/20: 3DVista has added Flash cubic output to its SHOW 2 software, and it appears to be good, but not cheap (EUR 449). This is a rather full-featured program that supports hotspots, etc. An example of its use by Aaron Spence is here. You can read further discussion of Flash in SHOW 2 here.
The equirectangular projection represents the entire sphere in a single rectangular image. Each unit of distance represents one degree of latitude or longitude. The image width represents the entire 360Â° of longitude, and the height the 180Â° of latitude, so the image aspect ratio is always 2:1. The entire top edge maps to the north pole and the entire bottom edge to the south pole. Because of the nonuniform nature of the mapping from equirectangular to rectilinear, this format presents the most difficulty for panorama viewing in Flash. The mapping is identical for all views with identical pitch (or latitude), so it’s easy to handle panning around with a displacement map, but it is different for different latitudes, so tilting up and down is a performance problem. On the demonstration page I show a rectilinear view of a low-res equirectangular image by Ben Kreunen from the Panotools Wiki. The displacement map is built into the swf because it would take too long to calculate in real time.
To get away from immersive panoramas for a moment, let’s look at the mapping of the equirectangular to one of the less weird non-immersive output formats, the orthographic projection. The orthographic projection represents a view of the sphere from infinity, hence without perspective, so locations in the view are easily expressible in terms of latitude and longitude. This is the fisheye lens view or the Christmas-tree ornament, depending whether you imagine it as concave or convex. In the center of the demo page I show an orthographic mapping of the same image. The calculation is much simpler and faster, so I haven’t stored it in the swf. And once it’s done, it can also be reused for any horizontally rotated view, as shown.
And here’s a great demonstration of different projections in action: Java World Map Projections. These views are all mapped in real time from the equirectangular projection.
What are the challenges for immersive panos in Flash? Performance, precision, and size.
Performance-wise, matrix operations and bitmap fills, such as those we can use to implement the cubic panorama, have the edge. In Java, it may be OK to keep recalculating pixel-based mappings, but Flash just can’t keep up. Therefore fully immersive navigation with the equirectangular format seems to be out of reach; I’d venture to say this remains true in AS3, although I haven’t tried it yet. Maybe you could precalculate 90 displacement maps, say, one for each degree of tilt, and use the appropriate one as the user navigates. Rather a large download I’d say. But that’s not the only problem.
The displacement map in Flash provides only one byte of precision for each of x and y, which means it can map a pixel value from one of at most 128 discrete locations on one side, and only 127 on the other. So when you map pixels over larger distances (which you can do using the scaleX and scaleY parameters of the DisplacementMapFilter constructor), you lose precision. In the equirectangular-to-rectilinear demonstration above, the y displacements are at most 110 pixels, but the x displacements go up to 195 pixels, so there’s already a slight mapping error compared to the original calculation. This would only get worse for larger images. It would have been nice if Macromedia/Adobe had allowed us to use two bytes for each displacement component, since the map image does have four bytes per pixel. I’ve tried to imagine some way of composing displacement maps to get greater precision, but I suspect it’s not possible.
Another limitation on Flash implementation of panoramas is the fact that Flash simply cannot load a bitmap larger than 2880 pixels in either dimension. This once again leads us towards the cubic format; since it uses six bitmaps, you can load them separately to represent a much larger pano than the other formats.