JavaScript's loose typing is a big benefit here because we are not burdened with a type system that is concerned about the lineage of classes. Instead, we can focus on the character of their contents.
-- Douglas Crockford (the MLK of JavaScript): JavaScript: The Good Parts
I’ve posted a cubic panorama example in the Sandy forum. If you haven’t seen the announcement yet, this awesome ActionScript 2.0 3D package has just released version 0.2. It’s early days yet for Sandy but already things are very impressive. Check out the features, the source, and the forum at Sandy’s Blog.
A note about texture mapping in Flash. I didn’t go into detail about the transformation-matrix method of texture mapping in my article Prospects for Immersive Panoramas in Flash. The idea is that since a single Matrix cannot perform a 3D perspective transformation, we implement it by slicing the texture into smaller pieces (generally triangles) and doing a matrix transformation on each of them. So there’s a tradeoff: the more pieces we cut the texture into, the better the result approximates the true perspective transformation, and the slower our movie runs. This is how Sandy does it, and you can see the technique abstracted into a standalone class by Thomas Pfeiffer here: DistortImage. This class gives you Photoshop-style free transform in Flash.
The DisplacementMapFilter is a filter that uses one bitmap to operate on another. The former bitmap is commonly called the mapBitmap; we’ll call it the map image.
Each pixel (x,y) of the output bitmap takes its value from an input image pixel specified by the color information of the pixel at (x,y) in the map image.
An output pixel with coordinates (x, y) is displayed according to the following formula. m[y][x], d[y][x], s[y][x] are the pixels at position (x, y) in the map image, destination image, and source image respectively.
var dx = (getComponent(m[y][x], componentX) - 0×80) * scaleX / 0×100;
var dy = (getComponent(m[y][x], componentY) - 0×80) * scaleY / 0×100;
d[y][x] = s[y + dy][x + dx];
function getComponent(color32:Number, component:Number):Number
{
switch (component)
{
case 1 : return 0xFF & color32 >> 16;
case 2 : return 0xFF & color32 >> 8;
case 4 : return 0xFF & color32;
case 8 : return 0xFF & color32 >> 24;
}
}
We show several special cases. We assume that neither componentX nor componentY is 8 (the alpha channel).
· Where the map image is 50% gray (0x808080), the output image is the same as the input image.
· Where the map image is black (0x000000), the output image becomes the input image shifted by (scaleX/2, scaleY/2).
· Where the map image is white (0xFFFFFF), the output image becomes the input image shifted by (-scaleX/2, -scaleY/2). (Actually a little less; the factor is really 0x7F/0x100.)
Place the cursor over any pixel in the output image [on the left] to see its source pixel in the input image [on the right].
Using the properties of the DisplacementMapFilter described previously, we’ll show how to make the map image.
We choose for our example the displacement needed to map an equidistant cylindrical [i.e. equirectangular] projection of half of the world map onto the globe.
Each pixel of the globe output image corresponds to a certain pixel in the input image. Let’s try to show that relationship by a formula.
This image shows the displacements of the lines of latitude and longitude. You can see that although the meridians become curved, the parallels remain straight. In other words, points with equal Y coordinates in the output image will correspond to points with equal Y coordinates in the input image. Therefore we can show the correspondence of Y coordinates without considering the X coordinates.
So first we consider the correspondence in the Y direction (latitude).
This figure represents the view of the output image from the left. The input image is mapped onto the curved line shown in blue.
Let’s try to show the formula that relates the input image Y coordinate (Y) and the output image Y coordinate (Y’).
In the figure, the Y arrow comes 1/3 of the way down from the top of the input image. This can be expressed as π/6 north latitude.
In this case the distance from the equator is sin(π/6), so the output Y coordinate (Y’) becomes (1 - sin(π/6)) / 2. Generalizing, we get
Y' = (1 - sin(π * (0.5 - Y))) / 2.
Now for constructing the map image the question is “for a certain output coordinate, what input coordinate does it come from?” So we solve the equation for Y:
Y = 0.5 - asin(1 - Y' * 2) / π
And from this we can derive what we really need for the map image, the displacement:
yd = 0.5 - asin(1 - Y' * 2) / π - Y'
We’ve derived the equation for the Y-axis displacement. Let’s make the filter:
import flash.display.BitmapData;
import flash.filters.DisplacementMapFilter;
import flash.geom.Rectangle;
Stage.scaleMode = "noScale";
var W:Number = 160;
var H:Number = 160;
var earth:BitmapData = BitmapData.loadBitmap("earth");
var map:BitmapData = new BitmapData(W, H, false);
var out:BitmapData = map.clone();
for (var y = 0; y < H; ++y)
{
var yr = y / H; // ratio of Y coordinate to image height
var lat = Math.asin(1 - yr * 2); // latitude
var yd = 0.5 - lat / Math.PI - yr; // difference between Y-coordinate ratios before and after transformation
var yc = Math.round(yd * 0x100) + 0x80;
map.fillRect(new Rectangle(0, y, W, 1), yc);
}
var disp:DisplacementMapFilter = new DisplacementMapFilter(map, null, 1, 4, 0, H);
out.applyFilter(earth, map.rectangle, null, disp);
attachBitmap(out, 0);
We can do the X coordinate transformation the same way.
This figure shows the output image viewed from above. The blue curve is the equator.
As before, we derive the equations relating the X coordinate (X) of a point on the equator of the input image to the X coordinate (X’) of the corresponding point on the equator of the output image, and their distance xd:
And since ew = cos(latitude), we refer to the way we calculated latitude last time and get
ew = cos(asin(1 - Y' * 2))
Thus we are able to express the (X, Y) coordinates of the corresponding point in the input image in terms of the (X’, Y’) coordinates in the output image.
Applying this to our previous ActionScript code:
for (var y = 0; y < H; ++y)
{
var yr = y / H; // ratio of Y coordinate to image height
var lat = Math.asin(1 - yr * 2); // latitude
var yd = 0.5 - lat / Math.PI - yr; // difference between Y-coordinate ratios before and after transformation
var yc = Math.round(yd * 0x100) + 0x80 << 8; // map's Y component
var ew = Math.cos(lat); // length of parallel
for (var x = 0; x < W; ++x)
{
var xr = x / W; // ratio of X coordinate to image width
var xd = Math.acos((0.5 - xr) / ew * 2) / Math.PI - xr; // difference between X-coordinate ratios before and after transformation
var xc = Math.round(xd * 0x100) + 0x80; // map's X component
map.setPixel(x, y, yc | xc);
}
}
var disp:DisplacementMapFilter = new DisplacementMapFilter(map, null, 4, 2, W, H);
However, since this code ignores the area outside the globe, that part of the input image remains unchanged.
Now outside the circle of the globe, (0.5 - xr) / ew * 2 is outside the range -1～1, and thus its arccosine is undefined. Let’s use this fact:
for (var x = 0; x < W; ++x)
{
var xr = x / W; // ratio of X coordinate to image width
var lap = Math.acos((0.5 - xr) / ew * 2); // longitude
if (isNaN(lap))
{ // undefined longitude means outside the circle
var xc = xr > 0.5 ? 0xFF : 0;
// set the map’s X component to a large number
}
else
{
var xd = lap / Math.PI - xr; // difference between X-coordinate ratios before and after transformation
var xc = Math.round(xd * 0×100) + 0×80; // map’s X component
}
map.setPixel(x, y, yc | xc);
}
And, in order to paint the pixel black when its source coordinates cannot be obtained, we modify the filter parameters:
var disp:DisplacementMapFilter = new DisplacementMapFilter(map, null, 4, 2, W, H, "color", 0x000000);
Image quality can be improved by using as much as possible of the 0x00～0xFF range of each of the map image components. I won’t go into detail on the procedure, but these are the main points:
1. Multiply the map’s X component by N with 0x80 as a center
2. Divide the map’s scaleX by N
If N is so large that the range 0x00～0xFF is exceeded, adjust it appropriately.
There’s a thriving community of photographers producing interactive 360° panoramas. Some cool immersive galleries are at panoviews.com and panoramas.dk.
Panographers are always interested in new ways to present the experience over the web. Currently the main vehicles are implemented in Java (e.g. PTViewer) or QuickTime (QTVR). There is also some use of Shockwave (Spi-V). But the idea of using Flash is attractive because of its ubiquity (thus no need to make the user engage in a lengthy download) and promise of cool features.
So what are the prospects?
Ordinarily in panorama viewing the target image for display is a rectilinear or gnomonic projection of the spherical field of view onto the viewing plane. This corresponds to the view through ordinary (non-fisheye) camera lenses and it’s the way people usually like to see pictures. As you will recall from looking at very wide-angle photos, distortion increases greatly at large angular distances from the center, but we generally accept it as natural. Other projections are occasionally used for artistic effects in panorama display; see Flemming Larsen’s remapping page. But for most people, the goal is to give the viewer the impression of being in the scene.
It’s also important to support navigation: the user should be able to pan and tilt through the environment, with the view updating in real time. So the software must continually remap the source image into the current view as the user interacts. Oh, and zoom too, but that’s easy.
Depending on the nature of the mapping, it can be calculated explicitly by analytic means, yielding a source pixel from which each destination pixel is to take its value, or it can be produced by means of matrix operations and bitmap fills on pieces of the input image.
Update 2006/1/28: The idea of the transformation-matrix method of texture mapping is that since a single Matrix cannot perform a 3D perspective transformation, we implement it by slicing the texture into smaller pieces (generally triangles) and doing a matrix transformation on each of them. So there’s a tradeoff: the more pieces we cut the texture into, the better the result approximates the true perspective transformation, and the slower our movie runs. This is how the Flash 3D package Sandy does it, and you can see the technique abstracted into a standalone class by Thomas Pfeiffer here: DistortImage. This class gives you Photoshop-style free transform in Flash.
In Flash, if we calculate the mapping on a pixel-by-pixel basis we have the option of storing the mapping in a displacement map and reusing it for any navigational position where it is valid. Calculating the mapping can be slow in Flash, but once it is done, applying it via a displacement map is very fast.
The source image is usually one of three types: cylindrical, cubic, or equirectangular. (For overviews of many many projections of the sphere, see MathWorld’s Map Projections or KartoWeb’s Map Projections.)
Of the three, the cylindrical cannot represent the entire spherical field of view, since to project a full sphere to a cylinder would require an infinitely tall cylinder. So it cannot provide the fully “immersive” pano experience where you can look up at the zenith and down at the nadir as well as all around; typically a cylindrical pano view allows only panning navigation and maybe a little tilt. However it is relatively easy to implement a cylindrical pano viewer in Flash, especially with a displacement map. Here’s Andre Michelle’s example (click on displacement->panorama_0; for a good time click on the other demos too). More detail on this in his blog entry.
The cubic projection is the one used in Quicktime VR, and consists of six square rectilinear faces, four representing the views in the cardinal directions, and two representing the up and down views. (In fact, if you watch the grid displayed as the image loads at panoramas.dk, you can see the edges of the cube.) Since the six faces are already rectilinear, if the viewer is facing one of them straight on, that face requires no remapping at all. The oblique views of the faces can be handled by the proper 3D rotation matrices. It will be trivial to implement this in Flash using a decent 3D package that takes advantage of Flash 8 features to map textures, such as the imminent 0.2 release of Sandy. In fact, Immervision has already demonstrated the technique directly without the use of any underlying 3D software. (They have said they have no current plans to make this a product.)
Update 2006/3/20: 3DVista has added Flash cubic output to its SHOW 2 software, and it appears to be good, but not cheap (EUR 449). This is a rather full-featured program that supports hotspots, etc. An example of its use by Aaron Spence is here. You can read further discussion of Flash in SHOW 2 here.
The equirectangular projection represents the entire sphere in a single rectangular image. Each unit of distance represents one degree of latitude or longitude. The image width represents the entire 360° of longitude, and the height the 180° of latitude, so the image aspect ratio is always 2:1. The entire top edge maps to the north pole and the entire bottom edge to the south pole. Because of the nonuniform nature of the mapping from equirectangular to rectilinear, this format presents the most difficulty for panorama viewing in Flash. The mapping is identical for all views with identical pitch (or latitude), so it’s easy to handle panning around with a displacement map, but it is different for different latitudes, so tilting up and down is a performance problem. On the demonstration page I show a rectilinear view of a low-res equirectangular image by Ben Kreunen from the Panotools Wiki. The displacement map is built into the swf because it would take too long to calculate in real time.
To get away from immersive panoramas for a moment, let’s look at the mapping of the equirectangular to one of the less weird non-immersive output formats, the orthographic projection. The orthographic projection represents a view of the sphere from infinity, hence without perspective, so locations in the view are easily expressible in terms of latitude and longitude. This is the fisheye lens view or the Christmas-tree ornament, depending whether you imagine it as concave or convex. In the center of the demo page I show an orthographic mapping of the same image. The calculation is much simpler and faster, so I haven’t stored it in the swf. And once it’s done, it can also be reused for any horizontally rotated view, as shown.
This mapping is the subject of a cool DMF Tutorial at the Japanese blog psyark.jp. Update 2006/01/16: I’ve posted a translation.
And here’s a great demonstration of different projections in action: Java World Map Projections. These views are all mapped in real time from the equirectangular projection.
What are the challenges for immersive panos in Flash? Performance, precision, and size.
Performance-wise, matrix operations and bitmap fills, such as those we can use to implement the cubic panorama, have the edge. In Java, it may be OK to keep recalculating pixel-based mappings, but Flash just can’t keep up. Therefore fully immersive navigation with the equirectangular format seems to be out of reach; I’d venture to say this remains true in AS3, although I haven’t tried it yet. Maybe you could precalculate 90 displacement maps, say, one for each degree of tilt, and use the appropriate one as the user navigates. Rather a large download I’d say. But that’s not the only problem.
The displacement map in Flash provides only one byte of precision for each of x and y, which means it can map a pixel value from one of at most 128 discrete locations on one side, and only 127 on the other. So when you map pixels over larger distances (which you can do using the scaleX and scaleY parameters of the DisplacementMapFilter constructor), you lose precision. In the equirectangular-to-rectilinear demonstration above, the y displacements are at most 110 pixels, but the x displacements go up to 195 pixels, so there’s already a slight mapping error compared to the original calculation. This would only get worse for larger images. It would have been nice if Macromedia/Adobe had allowed us to use two bytes for each displacement component, since the map image does have four bytes per pixel. I’ve tried to imagine some way of composing displacement maps to get greater precision, but I suspect it’s not possible.
Another limitation on Flash implementation of panoramas is the fact that Flash simply cannot load a bitmap larger than 2880 pixels in either dimension. This once again leads us towards the cubic format; since it uses six bitmaps, you can load them separately to represent a much larger pano than the other formats.
RSS
