Real‐Time Interactive Stereoscopy

• This is an age when science and technology looks upon itself, or is looked upon, as having a great influence upon art and artists.1 However, throughout history art and science have been so intertwined that any attempt to discover which has influenced the other is extremely difficult and speculative. However, on some splendid occasions a contribution from one culture to the other culture is clearly and distinctly made. Such an occasion occurred during the Renaissance when the Italian painter Leone Battista Alberti expounded in an essay on painting, usually referred to as the della Pittura, the earliest known geometrical scheme for depicting objects in a unified space, or what is today known as “perspective.”2 Most artists are thoroughly familiar with simple graphical techniques for using perspective in their work. Similarly, geometricians are thoroughly familiar with the simple equations, derived from nothing more complex than similar triangles, that characterize and define perspective projection from three dimensions to two dimensions. Since modern day digital computers are unable to manipulate straight edges and other drafting tools, this mathematical formulation of perspective geometry was an absolute necessity for using computers and automatic plotting devices for calculating and plotting perspective projections of three-dimensional data. Similarly, as soon as computer-generated movies were discovered, it was only natural that perspective was added. This perspective greatly enhanced the naturalness of these early computergenerated movies.3 Man is a creature inhabiting a world of three spatial dimensions jointly with his computer progeny. Effective communication between man and the computer should involve the manipulation and stereoscopic graphic display of three-dimensional data. This paper describes a real-time interactive stereoscopic computer facility with stereoscopic displays for output from the computer and a device for three-dimensional input to the computer. Man experienced the sensation of depth because he has two eyes, which view the world from two slightly different locations. Thus, if two slightly different perspective pictures are presented separately to each eyes, the human brain will translate their minute differences into a very realistic depth effect. As a computerized technique, the computer need only calculate two perspective projections of of the three-dimensional data, from two different viewing points. Pioneering work in the production of computer-generated stereoscopic pictures was done by Bela Julesz for stimuli in psycho-physical investigations of depth perception. Computergenerated stereoscopic pictures and movies were later used rather extensively to present scientific and technological data.5-9 As another aspect of computer graphics as a new form of man-machine communication, computergenerated images were being displayed in real-time on cathode ray tubes. Elaborate programs were written to permit the manipulation of images in an interactive manmachine environment. 10, 11 The marriage of stereoscopic computergraphics with real-time interactive computer-graphics was a natural and obvious union. There were, however, a number of problems in terms both of hardware for input and output of three-dimensional data and of appropriate software for quickly and easily manipulating three-dimensional data, and for calculating the final stereo pair of images. This paper discusses the compromises and final solutions of these problems and which made possible the actual implementation of a stereoscopic real-time interactive computer-graphics research facility. Descriptions are given of some of the practical uses of the facility and also some of the research work presently being conducted and planned for the facility. The derivation of the simple equations for stereoscopic projection are given in the appendix. A stereoscopic display presents the left and right images separately to the left and right eyes. There are many ways to accomplish such a separate presentation. This section of the paper discusses some of the more common methods. The old stereopticon consisted of two lenses and two pictures placed about 2.5 inches apart. Borrowing upon the simple concepts of this old method results in a computergenerated display counterpart as shown in Fig. 1. Two miniature, 1- inch diameter, cathode ray tubes separately display the left and right images. The viewer observes the faces of the two tubes through a simple lens system which has provisions for adjusting the focus and the interpupilary distance of the lenses. The small tubes provide high resolution (about 1000 lines per inch) and fast deflection speed, so that a reasonable amount of data can be presented flicker free. However, high voltages are necessary for the deflection plates of the tubes, and these voltages are quite close physically to the viewer, Apparently, the safety problem does not appear, since two miniature cathode ray tubes are being used on a helmet-mounted stereo system designed by Bell Helicopter Company for use by helicopter pilots and later adopted for computer graphics displays. 12, 13 A dual cathode ray tube display was constructed at Bell Labs using miniature tubes manufactured by Thomas Electronics (Fig. 2). The computer outputs the left and right images to both tubes. A blanking signal is used to turn off one of the tubes so that each eye sees only its appropriate image. One disadvantage of this system is the added electronic circuitry required for the blanking signals and the high deflection voltages for the cathode ray tubes. Also, a sizeable expenditure is required for a system which can be used only for three-dimensional computer-graphics. A single large-screen cathode ray tube can be used for stereoscopic displays by plotting the left and right images side-by-side on the same screen. Since the two images are now widely separated, prisms or mirrors have to be used for viewing the images as shown in Figs. 3a and 3b. This mirror system is normally used in those applications requiring exactness and careful control of image presentation, because with the mirrors, both images are viewed perpendicularly. However, since mirror systems tend to be large and bulky, the prism system seems best as a general purpose stereoscope. Even though the images are not being viewed perpendicularly, the viewer does not usually notice any image distortion. A simple prism stereoscope can be made and is a light weight attachment that can be easily slipped over the face of the cathode ray tube, as shown in the photograph of Fig. 4b. A mirror stereoscope is shown in the photograph of Fig. 4a. The last method of viewing stereoscopic images does not involve any stereoscope at all but is based upon the ability of the viewer to interrupt the normal relationships between his visual functions of convergence and focusing. In Fig. 3c, the viewer simply looks cross-eyed at the screen. At first this task is very difficult, but practice seems to help. Also, three images are seen, and it is the center image which must be focused upon. However, although this “method” is the simplest, its expense is in terms of the headaches and the frustration of the inexperienced as they attempt it! If a 3-D, interactive, computergraphics facility is to be useful, then the user must be able to communicate in three-dimensions with the computer. This means that a device, allowing effective and efficient input of three-dimensional data, must be available. This section of the paper discusses a few such three-dimensional input devices. The simplest, and in this case, the worst, 3-D input device uses three knobs to determine the X, Y, and Z input values. However, past experience demonstrated that two knobs for two-dimensional input to the computer are awkward to use, while such devices as light pens and pen-tablets are infinitely better. So adding a third dimension, in the form of a third knob, only makes more difficult an already miserable system. Hence, attention must be given to some pointer-like, or sticklike, or pencil-like device that controls all three dimensions at once so that one hand can be used to “draw” the three-dimensional data. As a possibility in this stick-like direction, consider the device shown in the sketch of Fig. 5. This device works in a spherical coordinate system. The two potentiometers mounted at the gimbal sense the angular position. A third potentiometer senses the position in the radial direction. The major problem working in this type of coordinate system is that the angular and radial position data must be converted to rectangular coordinates, since in nearly all applications computations are most simply performed by the computer in a rectangular coordinate system, for example, stereoscopic projections. This conversion would have to be made by the computer and would be time consuming enough to interfere with normal computational efficiency. Fig. 6 depicts a system using mutually orthognonal linear potentiometers to sense directly the position of the stick in rectangular coordinates. This was constructed and is shown in Figs. 7a and 7b. Since easy movement in three dimensions was critically important, the moving mass had to be as low as possible. Hence, aluminum and other light-weight materials were used. Thomson linear bearings were used to keep friction to a minimum. This simple-minded, but most effective, device has been in use now for about three years and has performed satisfactorily. All the applications described later in this paper have used this device for three-dimensional input to the computer. Figure 8 is a photograph of the 3-D input device used with the prism stereoscope. Other three-dimensional input devices have been reported in the literature, most notably the Lincoln Wand which used arrays of microphones to sense a high-frequency pulse.14 Timing information is used to fix the position of the wand within a three-dimensional space. Unfortunately, this device is subject to spurious reflections of the pulses and also requires somewhat complex electronic hardware. It is nevertheless an interesting gadget. The prime philosophy in the software package for stereoscopic computer-graphics was ease of use, with as simple a data structure as possible. Essentially the software package was to be user-oriented with simple FORTRAN-callable subroutines, with simple argument strings. The final software package actually consists of two groups of subroutines: one group for stereoscopic projections and a second group for general-purpose graphics. This section of the paper gives some feel for these subroutine packages. The general purpose graphics package has subroutines for drawing dots, lines, and characters at specified places on the display screen. The lines are drawn as a series of closely-spaced dots, which are interpolated between the end points of the lines. The interpolation is in terms of equal intervals along either the x-axis or the y-axis depending upon the slope of the line. The characters are produced from a basic 5 by 7 dot matrix and can be varied in size. All of these subroutines result in sets of x-y packed coordinate pairs being placed in specified output arrays. Other subroutines specify the names and sizes of the output arrays and turn the display interrupt on or off. The programmer deals with his data in terms of × and y coordinates, while the actual output data is packed for him by the subroutines. Hence the data structure is simply arrays of × and y coordinates while, the names of these arrays and their sizes form the bulk of the subroutine argument strings. Of course, everything is FORTRAN-compatible and the complete package requires about 600 (decimal) locations. The stereoscopic package is also FORTAN compatible, consists of two subroutines, and requires about 180 (decimal) locations. The threedimensional data must fall within a cube centered about the origin and with an edge length of 4095. The first subroutine is for initialization and for specifying the interfocal distance D and the viewing distance R. From these parameters, the scaling factor S and origin offset correction factor D/(2R) are computed. The second subroutine performs the stereoscopic projections from a single array containing the X, Y, and Z coordinates in sequence into three arrays containing XL, XR, and YLR also in sequence. Sense switches on the console of the computer are used to specify whether the left and right images are to be displayed on the single large cathode ray tube, or on the dual miniature cathode-ray-tube device, and whether the left image or right image is to be plotted on the left or right side of the display. This information is used by the stereoscopic projection subroutine to calculate the appropriate constant offsets so that the values of XL, XR, and YLR can be used directly without further modification in the general purpose graphics subroutines for actual final plotting. All of the preceding hardware and software was implemented on a Honeywell DDP-224 computer with 32,000 words of core storage. This computer is a 24 bit machine with floating-point hardware and a cycle time of 1.9 micro-seconds. The actual hardware and software details of this facility have been previously described.15 In addition to its use as a research facility for investigations of new forms of manmachine communication, the computer facility is also being used quite extensively for on-line speech research.16, 17 Speech scientists are quite accustomed to spectrographic displays of speech in which time is plotted along the X-axis, frequency is plotted along the y-axis, and spectral energy is plotted as a darkening of the paper. If the spectral energy were plotted in a third spatial dimension and if short-time spectral slices were calculated by the computer, then a three-dimensional plot of the speech spectra could be made. This was done before, but the plot was not very successful in making the spectral peaks or formats more apparent.18 Speech spectra were hence displayed stereoscopically with provision for rotating the spectra so that they could be viewed from different directions. In this way one might look down the valleys between the spectral peaks and better distinguish these peaks. Although the technique worked, no new insight was gained about the speech spectra. In another application, the facility was used to display stereoscopically the three-dimensional perspective projection of a four-dimensional hypercube.19 The stated purpose of the experiment was to determine if real-time interaction in the form of rotation in four-dimensional space might help to give some feel for a fourth spatial dimension. Unforunately, once again no new insight was obtained. Using an interactive computing facility for simply rotating threedimensional data is a waste of computing power since such rotation is best done with computer-generated rotational holograms.20 The computer is needed when calculations and interaction other than three-dimensional rotation is required. As an example, the display of the results of three-dimensional statistical scaling calculations normally would involve nothing more complicated than rotation, and hence a computer-generated rotational hologram would suffice. However, when a considerable number of data points must be displayed, each point cannot be individually identified since this would clutter the display. In an interactive facility, the user can use the three-dimensional input device to latch on to a point, and this point can then be identified by a number of some other information. Also, the display might be recalculated for only a selected subset of the data points. The full power of interactive three-dimensional computer graphics was required for a program for drawing in three three dimensions. Straight lines and curves could be drawn by using the three-dimensional device. These lines and curves could then be latched onto and either translated or rotated. All of this was real-time and only when extremely-complicated objects were drawn did a noticeable delay in rotation or translation occur, and for these cases, flicker was already also becoming noticeable. A scanned-type display was also available, and a stereoscopic software package was written for use with it. Although this scanned display had such a low resolution that only a little over 100 points were available for each separate image of the stereoscopic pair, a reasonable depth sensation was nevertheless possible. A stereoscopic scanned display would be particularly useful for presenting detailed surfaces stereoscopically without flicker although research in this direction has not yet been conducted using the DDP-224 computer facility. Over the past few years computer systems have been designed that use special digital hardware for performing rotations and perspective projects.22 Also, fairly elaborate and sophisticated programming languages and data structures have been devised for interactive computer graphics. 23 The computer system described herein does the rotation and stereoscopic-projection calculations by software written in assembly language for use as subroutines in FORTRAN or in assembly-language main programs. These software calculations are sufficiently fast to insure real-time operation, and the data structure and argument strings are sufficiently simple to insure programming ease. One can only wonder whether the special-purpose hardware was perhaps a needless complication and expense for most applications and whether the elaborate programming languages and data structures might not have actually impeded many application oriented uses. Most certainly the final responsibility is upon the user to make use of the most appropriate and expedient techniques for his particular application. Also, many of the special purpose hardware, sophisticated programming languages, and elaborate data structures are most interesting as research entities which must be adequately explored and evaluated for application to extremely complex interactive display situations. Other than three-dimensional drawing program, real-time interactive stereoscopy would seem to have been nothing but another academic exercise in man-machine communication. In most data display applications, the monocular perspective depth clues are sufficiently strong that stereoscopy can add very little more. For this reason, stereoptical data displays either interactive or non-interactive have not been widely used. The data must be devoid of monocular depth clues and reasonably complex before stereoscopy can be of assistance. Hence, the prime areas of use thus far are in design and in the visual arts. 24 However, the display of data involving four or more dimensions in such a way that the user can easily and quickly grasp the structure of the data is an unsolved and most challenging problem. Stereoscopy might be of critical importance in the solution of such display problems. The work described in this paper is really a small subpart of a project to develop a tactile man-machine communication facility, and it is within this that interactive stereoscopy will be of The tactile device will be used to the stereoscopic display for such as on to a or in three dimensions. It will also be used in investigations of between the human tactile and visual communication A tactile communication facility the to a man-machine communication and quite is a of its paper which was in The circuitry for the dual cathode-ray-tube stereoscope was designed by while the subroutines for characters were written by The design of the and the 3-D input device was performed by The DDP-224 computer facility with all its hardware and software was constructed by and by the and of Although the equations for stereoscopic projection are simple and easily they will be given since the of terms used in the equations might not be familiar and will be used mathematical techniques can be used to the The method the geometry of the and and nothing more complicated than similar a coordinate system X, Y, and Z with a point in this system by the coordinates and This point is to be on to the left and the right with viewing points and by the distance D and at a distance the The viewing points are at a distance from the The projection of and on to the left and the right the in the coordinate The focus distance in these equations as a constant all three coordinate values. Hence, it will only the of the left and right images. If the final images are to be in to the available plotting space, then the focus can be since it will only be by some other This paper is in of the for the of at the of is on the of the of and of the of the a he to from years with Bell he was early with computer and investigations of short-time and the method for also computer-generated displays of data, application of computer technology to the visual and investigations of human to At the time he left Bell Labs in he was more effective forms of man-machine communication, real-time computer graphics and tactile communication. has been widely and his has been throughout the world and shown on four in automatic speech is the of and is a of