Remote Reality is an approach to providing an immersive environment via the use of omni-directional imaging. The remote reality environment allows the user to naturally look around, within the hemispherical field of view of the omni-directional camera, and see objects/actions in that view. It can use live or pre-recorded video from a remote location. While less interactive than traditional VR, remote reality has important advantages: there is no need for "model building," and the objects, textures, and motions are not graphical approximations.

This document will provide an overview of past, present, and future work in remote reality at Lehigh University, see also C. Power's Remote Reality Page . We begins with an overview of omni-directional imaging. The second section describes the current remote reality system and smaller, more portable designs we are looking at. The next section focuses on applications of remote reality currently under development. Finally, the document ends with a few example video's, and some brief comments about and a link to our Teleoperation Experiment. In addition, you can take a look at some remote reality pics in the photo gallery.

Omni-directional imaging refers to the ultimate wide-angle lens, a system that can capture an image in every direction. Over the years many such systems using moving cameras or "fish-eye" lenses have been developed to generate "approximate" omni-directional images. They are only approximate because they do not have a single perspective "viewpoint", but a collection (finite or in the case of most fish-eye lenses, an infinite locus of points). The newest systems, which use special devices to attempt to rotate about the "perspective point" of a lens attempt to reduce the distortions caused by the varying viewpoints, have been greatly improved for construction of omni-directional "static" images. Setup and "calibration" of these cameras, as well as controlling movement in the area being imaged, requires care and planning.

The development of the Remote Reality system was made possible by the recent research of S. Nayar, which revolutionized wide-field of view imaging by introducing the paracamera (used interchangeably with omni-camera in this document), a system that directly captures a full hemisphere (or more) while maintaining a single perspective viewpoint. (see Columbia's CAVE web site for more info on Omni -Directional imaging, Cyclovision's web site for details on commercial omni-directional cameras, and our omni-directional VSAM effort. (joint with S. Nayar Columbia Univ)

Because the Paracamera captures the viewing hemisphere (or more) simultaneously, it can be used for full motion video. Furthermore, placing two paracamera systems back-to-back allows a true viewing sphere, i.e. 360 x 360 viewing. Unlike fish-eye lenses, each image in the paracamera system can be processed to generate GEOMETRICALLY CORRECT perspective images in any direction within the viewing hemisphere.

The omni-directional imager combines an orthographic lens and a parabolic mirror, where the axis of the parabolic mirror is parallel to the optic axis. Because the lens is orthographic, entering rays are parallel. By definition, rays parallel to the axis reflect off a parabolic surface at an angle such that they virtually intersect at the focus of the parabolic surface. Thus the focus of the paracamera provides a single "virtual" viewpoint. The single virtual viewpoint allows for consistent interpretation of the world in any viewing direction. (see accompanying diagram)

To see how we generate a proper perspective image from the para-image, consider an "imaging array" in the desired viewing direction. For each pixel, logically cast rays toward the focus and follow them as they "reflect off" the parabolic surfaces focus and follow them to where intersect the measured image. The resulting spatially varying resampling can be very efficiently implemented using spatial lookup tables. In the remote reality system, when turning the HMD (or giving commands to zoom-in or zoom-out), the virtual viewpoint is stationary; only the location of the virtual "imaging array" is moved. Thus the transitions between views is very very natural.

While we are describing the remote reality system using a HMD and head tracker to control viewing, we also note that the system could (and the commercially available software from Cyclovision does) use a mouse or joystick to control viewing and the window in a regular computer display to show the resulting unwarped image.

The main components of the remote reality system are the omni-directional camera, video recording systems, PC computing platform, head-mounted-display (HMD), and head-tracker. The current system balances cost and quality. Our current data collection system was approximately $5K and the computing/HMD/head-tracker play-back system was about $3K. Prototypes using a wearable computer are currently under development(see below for more details on those).

Because omni-directional imaging compresses a viewing hemisphere into a small image, maintaining resolution and captured image quality is quite important. While the process scales to any size imager, the current system uses NTSC (640x480) or PAL (756x568) cameras. Note the "spatial resolution" of the image is not uniform. While it may seem counter intuitive, the spatial resolution of the omni-directional image is GREATEST along the horizon, just objects are most distant. The spatial resolution along the horizon is 4.2 pixels/degrees (5.1 for PAL). While we have experimented with many capture systems, the best quality to date has been obtained using the Cannon Optura (which is about half the cost of the collection system). While the Cannon is a bit more expensive, it has a progressive scan mode, which removes problems with interlacing issues. As the camera is generally moving for remote reality applications, the motion between the odd and even frames is quite noticeable in the processed image. (We hope to add examples showing the difference)

The playback system uses a 200-300MHz MMX CPU (running Linux), and a video capture card. The system computes monocular SIF-resolution full-rate "video" (320x240@30 fps NTSC) which is reasonably well matched to the Virtual I-O glasses. The built in head tracker of the I-O glasses, provides yaw, pitch, and roll, with updates to the viewing direction at 15-30fps. A mouse or joystick can also be used for view selection. A lower speed CPU could be used for playback at 30fps, but cannot update the viewing direction at that rate.

Wearable/Portable Systems
While the current implementation still uses a desktop CPU, we are currently building

Wearable System

several smaller versions of the system that range from wearable to very portable. The wearable version (seen at right) is a PC-104+ based system that uses a Pentium MMX CPU and fits (not including batteries), in a 5"x5"x5" full-cibe enclosure (We had a half-cube system but it could not handel@30fps video). The unit is equipped with an 2G hard disk with Linux/X Windows installed, a video capture board, wireless networking, and VGA display output. Power for the unit is supplied by inputting 9 - 30 volts to a Power Trends 6302 ISR regulator, thus allowing us to power the wearable with various types of power supplies. Currently we use either a standard AC brick supplying 12 volts, (2) Sony NP-F730 Lithium ion batteries in series supplying 14 volts, or (2) Ni-Cad batteries in series also supplying 14 volts. For display we use either the i-glasses HMD, or a monocular HMD.

Understanding the user interface issues of immersive video and wearable computing in a subject of our ongoing work. Mpeg video's of some of the different interfaces we are exploring are:

A biocular (true Remote-reality) HMD which is suitable for a driver in a tele-operation setting. This shows an older i-glasses based system. We now use Sony Glasstrons HMD with Intersense head-trackers . The user points via their head direction, which combined with the dual display makes it very immersive but walking is difficult.

A monocular HMD suitable for a follow up team member in a MOUT scenario or for general remote collaboration. This shows a liquid image M1 unit where view selection is head-tracked (or a full 360 view is presented directly). Head tracking leaves both hands free for driving. We also allow view selection via trackball or a twiddler. A useful alternative is the dual-panoramic display, whichwe call it the driving view , because one of the two panorams is flipped like a rear-view mirror.

A "computer free" interface video broadcast to a hand held TV or it can be directly displayed on an HMD. This allows the user to see the whole area but does not require a body-worn computer. (Which are currently expensive and heavy).

In addition to the wearable system, two very portable systems are also in development. By very

Netwinder275

portable, we mean a system that can be relocated easily. The first of these two was chosen because of the large number of features it offered in a small, tightly integrated package. It is the Netwinder275 from Corel Computer Corp (netwinder.org), a StrongARM based Linux box. In this small box, measuring 9.5"H x 6"D x 2"W, the Netwinder has a 275MHz StrongARM processor, 32 or 64MB ram, an 810MB hard disk, 2 Ethernet connections (10 BaseT and 10/100BaseT), a parallel, serial, and IrDA port, 2MB video adapter, 16-bit sound, built in microphone/speaker, and most important to our project, NTSC/PAL composite video in/out via an on-board video capture/playback device. The performance of the Netwinder is suprisingly good, as we can currently capture and unwarp omni video in 320x240 at 30fps. All of these features integrated into such a small well-performing package make the Netwinder a solid choice for a relocatable remote reality system.

The second portable remote reality system we have in development is designed primarily for use by

Ranger - A rugged
CompactPCI system

the military. Made by Kinetic Computer Corp. (kin.com), the Ranger (.pdf file) is a very rugged unit that was designed for high shock, high vibration, exposed environments. The unit itself ways approximately 10lbs., and is made of black anodized aluminum. The computer inside the Ranger is based on a Gespac CompactPCI PCISYS-56 system using a Pentium processor, 64MB EDO RAM, a complete set of I/O ports/interfaces, a 16-bit PC-104 extension port, and a SVGA controller for CRT's and LCD's. The entire system sits on a Compact-PCI backplane, supporting up to 3 cards, in addition to the motherboard. We utilize 2 of these 3 slots for a 10BaseT ethernet card, and a video capture card. Sitting on the PC-104 bus is a PCMCIA module card with 2 PCMCIA slots. One of these slots is currently occupied with a PCMCIA wireless networking card.

More recently our efforts have been focused on developing applications of the remote reality system. Its applications in entertainment are rather obvious. More direct applications include training and mission rehearsal (for military actions), real-estate sales, education, and remotely piloted vehicle operation. The later, remotely piloted vehicle operation, also known as teleoperation, is one of our main areas of interest.

The prototype system (seen at right) was developed in hopes of proving the feasibility of the technology using a cheap off the shelf R/C car and other commercially available products. Several areas are of main

Prototype

concern when developing a vehicle to be remotely piloted. The most obvious is the vehicle and its control mechanisms. Others include the camera and video transmission system, as well as methods used to dampen vibrations and EM interference.

The vehicle itself is a cheap ($150) R/C car purchased for its large size (16"L x 12"W x 10"H) and ground clearance (approximately 6"). The large size enables it to be a stable camera platform, and the high ground clearance is useful for clearing small (2-3") objects. To control the car, the radio transmitter/receiver pair that came with the car were used. They operate at 27MHz and have a range of about 50-75 ft. Two different camera systems were tried on the prototype to evaluate their quality, geometry, and weight. The camera system seen at right is an early omni-cam prototype that uses a Panasonic Digital Color CCD camera. These cameras are quite heavy and the car's performance was severely degraded as a result. The camera system used predominantly for early testing was a commercial version of the omni-camera available from Cyclovision. It is slightly lighter, has better weight distribution, and worked fairly well even in low light situations. For video transmission, we used a transmitter and receiver we happened to have in the lab already. The system is a 2.4GHz based Tx/Rx pair that can operate on four different user-selectable channels. It requires 12 volts DC and for the purposes of testing the vehicle we chose to use (2) 7.2 volt R/C batteries wired together is series. These batteries added significantly to the weight, but were able to be quickly recharged.

The results of the initial tests on the vehicle were promising. We found that it was possible to effectively pilot the vehicle remotely. Early tests showed the advantages the omni-cam provided over traditional forward looking cameras. Our results were however not all positive. We found the effects of vibration and electro-magnetic interference from the car's DC motors and tx/rx equipment to be significant. We did however prove enough feasibility to warrant further testing, See the Teleoperation Experiment section for more information.

Para-RROVer
Parabolic Remote Reality Omni Vehicle

The Para-RROVer
Due to the significant problems associated with the various aspects of the prototype's performance, it was necessary to redesign all of the major systems of the vehicle for further testing. The second generation prototype, known as the Para-RROVer or Parabolic Remote Reality Omni Vehicle, is a completely new design with the exception of the frame, tires, and motors. The frame and tires were kept because of the size and ground clearance they provided, and the motors were kept because they provided a good mix of speed versus runtime. Everything else on the vehicle has been upgraded for a variety of reasons that will be discussed next.

The first thing to be replaced was the tx/rx system that came with the vehicle. The range it provided, 50 - 75ft., was completely inadequate for the purposes of teleoperation. It was replaced with a Futaba Skysport 4-channel FM system. This system not only increases the useable range of the Para-RROVer to over 4000 feet, it adds two control channels for possible future addition of a pan/tilt unit. Another major system that was replaced was the video transmitter and receiver. The original 2.4GHz system worked well, but was limited to line of sight transmission between the transmitter and receiver. This meant that the signal could not travel through walls, doors, or even humans, which makes for a poor transmission system for teleoperation. We found we

Hand Held TV Receiver
for 434MHz Tx

needed a slightly more powerful system that was lower in frequency to enable the signal to penetrate objects better. We found a system that meet those requirements from Extreme Camera Systems. The 900MHz tx/rx system is not only significantly lower in frequency, but has an advertised operating range of approximately 4000 feet, which goes well with our Futaba transmitter. Another benefit this system offered was its power supply. It has a 9-volt connector, which allowed us to plug in the 8-AA, 12-volt pack that it came with, or use a standard 9-volt battery in emergencies. In addition to the 900MHz system, we are also experimenting with a 434MHz system from Extreme Camera Systems. In addition to its lower frequency, 434MHz is UHF channel 13, and therefore can be received using a simple TV antenna. As a result, the video feed can be received using one of the many COTS TV tuners available today. Currently we can use either a standard TV/antenna, or a small hand held TV with built-in tuner. The hand held TV has the advantage of being very small, lightweight, and easily portable.

Another important component to be replaced was the camera. While the commercial version of the omni-cam being used was quite adequate, in that it was light, fairly sturdy, easily mountable, and had good low light characteristics, it had its drawbacks. The commercial version provided only modest quality video, was interlaced and therefore suffered from issues in that area, and had no means for on vehicle recording, something we determined was very important when performing experiments. The camera we chose to replace it was the Cannon Optura. It offers two main advantages over the previous cameras: (1) it is 640x480 progressive scan (non-interlaced) which gives better quality video with less interlacing effects, and (2) it can record the video on mini-dv's whereas the previous camera had no internal recording mechanisms. Another advantage is it uses a folded optics system (designed by T. Boult, L.U.), meaning the light reflects off the parabolic mirror, then off the planar mirror and into the camera's lens. The folded optics system allows the

Optura Camera Mount

camera to sit below the mirror, and thus lowers the center of gravity of the unit and makes it less obtrusive. The use of the Optura does have its disadvantages, one of those being added weight. The Optura is approximately 2 lbs. heavier than the color version of the paracamera from Cyclovision, and 3 - 4 lbs. heavier than the black and white version. The mounting is also somewhat complicated due to the akward angle at which the Optura must be placed. With the previous camera, velcro was the only mounting equipment neccesary, and served not only to mount the camera, but to reduce vibrations transmitted to the camera. With the Optura, a new mounting scheme must be used, one that holds the camera and serves to reduce vibrations as the velcro previously did, this also increases the weight. We have gone through many designs, too many to mention here, and are still experimenting to try and find just the right mount. One of the currently used mounts(seen at right), does a good job of mounting the camera by keeping it's CG close to the ground, but does a poor job of limiting vibrations. Also, due to flexing in the aluminum bar used for mounting, it allows the entire camera assembly to torque forward and backward during starting and stopping. We are currently experimenting with other mounts including a ball and socket style mount, as well as vibration reducing rubber.

One of the problems we encountered when adding the weight of the Optura was poor shock performance. The standard shocks the vehicle came with were never designed to withstand the heavy load being placed on them. They were replaced with heavy duty adjustable oil-filled aluminum shocks. These shocks have stiffer springs and provide better dampening than the original shocks. One final thing that must be mentioned is the effect of EM interference on the video signal. The EM interference emitted by the vehicle's electronics, as well as the Cannon Optura, is severe and must not be overlooked. Not accounting for this interference severely limits any video transmission system placed on the vehicle, no matter what it's capabilities. To combat this, we use proper placement of the transmission antenna, and EM interference blocking materials. We have tried mounting the transmitter several places on the vehicle, and the best to date has been the highest point. Placing the transmitter at the highest point achieves two things. First, it gets the transmitter as far away from any interference as possible, and second, no objects on the vehicle ever block the path from the transmitter to the receiver (unless of course the vehicle is above in elevation relative to the receiver). To block some of the interference, we rely on combinations of wire mesh caging, and aluminum foil. While that may sound crude, we have found they are good, inexpensive solutions to the problem of interference.

The Para-Chopper
In addition to the Para-RROVer, we are currently in the design and development stages of what we call the Para-Chopper. The Para-Chopper is an R/C helicopter designed to extend the concept of remote reality to the sky. It is the next step in omni-cam based teleoperated vehicles. Interested parties should look for the protoype mid 1999.

The Underwater ParaCam
In addition to mounting the paracamera on cars and helicopters, we have been developing an

Second Generation
Underwater ParaCam

underwater omnicamera. The first prototype used a Hi-8 camera (lots of interlacing issues), and while functional, it was mostly intended to allow us to experiment with various design issues. The second generation camera(a Cannon Optura) is considerably more compact, and due to its progressive scan mode, does not suffer from interlacing effects. Note that in the second design we used optical folding (a second mirror) so the system only protrudes about 8 inches from the camera. It also allows the camera to "view" in the outward direction, i.e. the omni-directional image is looking, more or less, the same direction as the viewing portal. There are no moving parts and because the camera's viewing portal is flat glass, it is ideal for deep water operations. Note that the flat glass portal is not an approximation, it is an ideal configuration for the paracamera design. The only difficult issue for deep water operation is getting sufficient light, which is a little more difficult for the paracamera since it means lighting a hemisphere.

Various aspects of the Remote Reality system have been demonstrated at the 1998 JASON's meeting, the DARPA 40th anniversary celebration, CVPR98, Siggraph 98, the 1998 VSAM Workshop, WACV98, and the 1998 DARPA Image Understanding Conference. It has also been shown to various military groups and S. Peleg had a demo system running at Hebrew University for a while.

This section has a few videos from the paracameras, as they might be used in remote reality. These videos are MPEG clips of the raw unprocessed output, meaning they are fixed view. (The processed output will not be available until we get a new VGA->NTSC converter, a student plugged the wrong power-supply into our last one :-(). The first example is a short clip from our "tour" of the Ft. Benning MOUT site. In this example the paracamara was mounted on the windshield of a car. The full tour is about 15 minuets, here we have a short (1 min) clip from the beginning. The front of the car is to the upper right in the video. With a little practice, one can get good situational awareness directly from the paraimage sequence, though a HMD with the unwarped video is easier.

As was mentioned maintaining resolution is very important . We point out that to keep the size of the examples down, we have generated SIF resolution MPEG's. The reduction from 640x480 to 320x240 is a significant loss. In addition, Mpeg adds a bit of bluring and noise. We will (soon), have some of the processed 320x240 unwarped "output" associated with these clips, but with the loss of user control much of the impact is lost as well.

Teleoperation, the remote operation of a vehicle or robot, has a growing list of applications. As the Para-RROVER demonstrates, the omni directional imaging sensor when combined with our remote reality system provides the tools necessary for teleoperation. For the we have a large experiment planned which will test the teleoperation capabilities of several interfaces to our remote reality system against that of a single wide angle lens. Click on the link above to read more about the experiment.