Small Vision System User Manual SRI Small Vision System User’s Manual Software version 2.3i September 2002 ? Kurt Konolige and David Beymer SRI International konolige@ai.sri.com http://www.ai.sri.
Small Vision System User Manual 2 1 Introduction ______________________________________________________ 5 1.1 The SRI Stereo Engine and the Small Vision System ________________________ 6 1.2 The Small Vision System_______________________________________________ 7 1.3 Hardware and Software Requirements ___________________________________ 8 1.3.1 1.3.2 Analog Framegrabbers ___________________________________________________ 8 Digital Framegrabbers ___________________________________________________ 8 1.
Small Vision System User Manual 3 3.4 Area Correlation Window_____________________________________________ 39 3.5 Multiscale Disparity__________________________________________________ 41 3.6 Filtering ___________________________________________________________ 42 3.7 Performance ________________________________________________________ 44 4 Calibration ______________________________________________________ 45 4.1 Calibration Procedure ________________________________________________ 46 4.1.1 4.1.2 4.1.
Small Vision System User Manual 5.8.2 4 Multiscale Stereo Processing _____________________________________________ 65 5.9 Window Drawing Classes _____________________________________________ 66 5.9.1 5.9.
Small Vision System User Manual 1 5 Introduction The SRI Stereo Engine is an efficient realization of an area correlation algorithm for computing range from stereo images. Figure 1 shows the results of running the algorithm on a typical scene. The image on the top left is the left image of an original stereo pair, while the one on the top right is a disparity image computed from the stereo pair.
Small Vision System User Manual 1.1 6 The SRI Stereo Engine and the Small Vision System The Stereo Engine exists in several implementations, including embedded, low-power systems and general purpose microcomputers. The embedded systems, or Small Vision Modules (SVMs), contain DSPs or other standalone processors, and produce digital range information. They are meant for end applications where size, cost, and power limitations are critical.
Small Vision System User Manual 1.2 7 The Small Vision System The Small Vision System (SVS) is meant to be an accessible development environment for experimenting with applications for stereo processing. It consists of a library of functions for performing stereo correlation. Figure 1-2 shows the relationship between the SVS library and PC hardware. Images come in via a pair of aligned video cameras, called a stereo head.
Small Vision System User Manual 1.3 Hardware and Software Requirements The SVS libraries exist for most Unix systems, as well as MS Windows 95/98/ME/2000/NT; that is, on the most common computer platforms available. We have spent considerable effort in optimizing SVS for PCs using the MMX instruction set (Pentium MMX and Pentium II, III, IV), and it will perform best on these platforms, using either Linux or MS Windows. Performance information is in Section 3.7. 1.3.
Small Vision System User Manual 1.4 The SVS Distribution The SVS distribution can reside in any directory; normally, it is placed in c:\svs (MS Windows systems) or /usr/local/svs or a user’s directory (Unix systems). Here is the directory structure of the SVS distribution. svs readme update docs smallv.pdf calibrate.pdf bin smallv(.exe) smallvcal(.exe) stframe(.exe) stcap(.exe) svsXXX.dll,.lib XXXcap.so svs.dll, lib svscap.dll, lib libsvs.so libsvscap.so fltkdll.dll libfltk.so data check.
Small Vision System User Manual 2 10 Getting started with smallv The smallv program is a standalone application that exercises the SVS library. It is a GUI interface to the stereo programs, and in addition can load and save stereo image sequences. The smallv program is a useful tool for initial development of a stereo application, and can also be used to check out and adjust a stereo camera setup. The smallv program is in the bin/ directory.
Small Vision System User Manual 11 are interested in using a particular framegrabber and set of cameras with smallv, please see Section 2.1. The framegrabber interface is indicated in the message area on startup. In this case, it is the MEGA-D digital stereo head.
Small Vision System User Manual 2.1 Inputting Live Stereo Video The SVS libraries provide support for live video as stereo input. To input video, you must do the following steps. 1. Decide on a stereo head and framegrabber. 2. Install the framegrabber, following instructions that come with the framegrabber or the stereo head (STH-V3 or MEGA-D). 3. Copy the appropriate framegrabber DLL to bin\svsgrab.dll (MS Windows), or bin/libsvscap.so (Unix); see Section 2.1.2). 4.
Small Vision System User Manual 1. Use two framegrabbers, and input one signal on each. 2. Use a single framegrabber capable of inputting 2 monochrome channels, e.g., the Matrox Meteor RGB or Meteor II / Multichannel. 3. Use a single framegrabber, and a stereo head that interlaces two video signals onto a single video stream. The STH-V3 from Videre Design (www.videredesign.com) is one such stereo head. 4. Use a digital stereo head, the MEGA-D (STH-MD1) from Videre Design.
Small Vision System User Manual There are convenient BAT files to set up a particular framegrabber, in the bin\ directory. Simply double-click on the batch file to perform the required copying. For example, to set up the Imagenation PXC200 framegrabber, go to the bin\ directory in Windows Explorer, and double-click on the file setup_pxc.bat. The framegrabber interface must be set up before starting smallv; it cannot be changed while the program is running.
Small Vision System User Manual Left camera field Single video stream Interlaced field Right camera field Figure 2-3 Line-interlaced stereo cameras (STH-V3 stereo head). Alternate lines from each camera are interlaced into a single video stream. Only one type of device, the MEGA-D or Dual DCAMs, will be seen by the smallv program. The choice depends on which interface library has been loaded (see Section 2.1.3).
Small Vision System User Manual A subset of frame sizes are supported for video input in the smallv application; the following table summarizes them. Most framegrabbers support hardware interpolation and scaling, so that bus traffic is minimized by working with smaller frames. The exception is the Meteor RGB, which passes a full field to memory, where it is decimated by the SVS software.
Small Vision System User Manual ? ? Freeze mode. In this mode, a single stereo pair is input, then the same frame is continuously processed and displayed. This mode is useful in checking the effect of different stereo parameters on the same image. 2.1.9 Adjusting Video Parameters Most framegrabbers support some kind of video image adjustments, such as contrast or brightness. The video parameter dialog is invoked using the Video… menu item (Figure 2-4).
Small Vision System User Manual 18 2.1.10 Subwindowing The MEGA-D digital stereo head can send to the host computer just a portion, or subwindow, of the stereo image. For example, if the MEGA-D is in x2 sampling mode (full-size image is 640x320), and the image size is chosen to be 320x240, then smallv will input only a 320x240 subwindow of the full image. Figure 2-5 shows two of these subwindows, and the original full-size image.
Small Vision System User Manual 19 2.1.11 Vergence When in subwindow mode, the two cameras in a stereo rig generally will have the same X and Y offsets, so that they keep the parallel line-of-sight characteristic of the stereo rig. However, for viewing close objects, it is advantageous to toe-in, or verge, the two stereo cameras. In this way, the images of the near object will both contain the object in the center. Human eyes verge mechanically when viewing close objects.
Small Vision System User Manual 20 2.1.12 Color As of Version 2.1, SVS supports color input and display. Besides the two monochrome left/right stereo channels, there is a third color channel that corresponds to the left image, with images in RGB 32bit format, and optionally a fourth color channel for the right image. The color channels do not participate in stereo processing, but can be useful in applications that combine color and stereo information, for example, object tracking.
Small Vision System User Manual 2.2 Storing, Saving, and Loading Stereo Data smallv provides a basic facility for loading and saving stereo data streams. The file load and store functions are part of the SVS library, and their source code is included. smallv exercises these functions, and provides a memory buffer for storing live stereo video. There are two basic types of storage/playback available.
Small Vision System User Manual 22 buffer, or individual images, can be saved to a file. Under the File menu, use the commands Store Video Buffer and Load Video Buffer. A video sequence is stored as a set of BMP files (next subsection), in a sequence starting at 001. You cannot save stereo disparity or 3D data directly from the video buffer. However, the current frame disparity and 3D data can be saved using the still image storage facility, described in the next subsection. 2.2.
Small Vision System User Manual 2.3 23 Display smallv displays two images in its display area. The left display is always the left input image. Input images are displayed in grayscale, unless color information is present: in this case, the left image will be shown in color. The right display can be either the right input image, or the results of stereo processing. Processing results are always displayed in “greenscale”, using shades of green.
Small Vision System User Manual 2.4 24 Stereo Processing and Parameters In smallv, stereo processing takes place in conjunction with the input of stereo images. The basic cycle is: get stereo pair -> process pair -> display pair The input is either from live video or the buffer (Sections 2.1 and 2.1.9). In freeze mode, the same pair is processed continuously, so adjustments can be made in stereo parameters. 2.4.
Small Vision System User Manual Figure 2-8 3D display window. The red ray is the optic ray from the left camera.
Small Vision System User Manual 2.4.3 26 Calibration For good stereo processing, the two images must be aligned correctly with respect to each other. The process of aligning images is called calibration. Generally speaking, there are two parts to calibration: internal calibration, dealing with the properties of the individual cameras and especially lens distortion; and external calibration, the spatial relationship of the cameras to each other.
Small Vision System User Manual 27 ranges enlarge the horopter, but not in a linear fashion, i.e., a search range of 32 does not give twice the horopter range of 16; see Section 4 for technical details. Changing the disparity search size affects the time it takes to process stereo. A search space of 32 pixels will take about twice as long as a search space of 16 pixels. It will actually take a little less, because there is some fixed overhead in processing the images.
Small Vision System User Manual 28 larger window will produce smoother disparity images, but will tend to “smear” objects, and will miss smaller objects. A smaller window will give more spatial detail, but will tend to be noisy. Typical sizes for the window are 9x9 or 11x11. The window size is selected using the Sum window drop list. In the MMX implementation, not all window sizes are supported. More technical information on the correlation window can be found in Section 3.4. 2.4.
Small Vision System User Manual 2.5 29 Filtering Stereo processing will generally contain incorrect matches. There are two major sources for these errors: lack of sufficient image texture for a good match, and ambiguity in matching when the correlation window straddles a depth boundary in the image. The SVS stereo processing has two filters to identify these mismatches: a confidence measure for textureless areas, and a left/right check for depth boundaries.
Small Vision System User Manual 2.6 Saving and Restoring Parameters All of the parameters that control the operation of the SVS Stereo Engine can be saved to a file for later use. Parameter files can be loaded and saved using the File menu: Load Param File and Store Param File. The file data/megad-75.ini contains a sample file for a 7.5 mm lens on the MEGA-D stereo rig. It serves as an example of the settings available through parameter files.
Small Vision System User Manual Tx Ty Tz Rx Ry Rz -89.458214 -0.277252 -0.923279 -0.008051 -0.003771 -0.000458 [left camera] pwidth 1280 pheight 960 dpx 0.007500 dpy 0.007500 sx 1.000000 Cx 582.260123 Cy 506.081223 f 7.798704 kappa1 0.002983 kappa2 -0.
Small Vision System User Manual 3 32 Stereo Geometry Stereo algorithms compute range information to objects by using triangulation. Two images at different viewpoints see the object at different positions: the image difference is called disparity. This section discusses the basic equations that govern the relationship between disparity and range.
Small Vision System User Manual 3.1 33 Disparity The figure below displays stereo geometry. Two images of the same object are taken from different viewpoints. The distance between the viewpoints is called the baseline (b). The focal length of the lenses is f. The horizontal distance from the image center to the object image is dl for the left image, and dr for the right image. Figure 3-1. Definition of disparity: offset of the image location of an object.
Small Vision System User Manual Figure 3-2. Inverse relationship between disparity and range. This plot is for a focal length of 6.3 mm, a baseline of 80 mm, and a pixel width of 14 mm.
Small Vision System User Manual 3.2 Horopter Stereo algorithms typically search only a window of disparities, e.g., 16 or 32 disparities. In this case, the range of objects that they can successfully determine is restricted to some interval. The horopter is the 3D volume that is covered by the search range of the stereo algorithm. The horopter depends on the camera parameters and stereo baseline, the disparity search range, and the X offset. Figure 3-3 shows a typical horopter.
Small Vision System User Manual Disparity: 5 4 3 2 1 0 -1 -2 -3 Figure 3-5. Planes of constant disparity for verged stereo cameras. A search range of 5 pixels can cover different horopters, depending on how the search is offset between the cameras. The location and size of the horopter depends on the application. If an object falls outside the horopter, then its true disparity will not be found, and instead it will get some random distribution of disparities.
Small Vision System User Manual 37 to arbitrary resolutions. These first three options change the camera geometry, and thus have a corresponding effect on the range resolution, which decreases (see below). The only way to increase the horopter size and maintain range resolution is to increase the disparity search window size, which leads to more computation.
Small Vision System User Manual 3.3 38 Range Resolution Often it's important to know the minimal change in range that stereo can differentiate, that is, the range resolution of the method. Give the discussion of stereo geometry above, it's easy to see that that range resolution is a function of the range itself. At closer ranges, the resolution is much better than farther ranges. Range resolution is governed by the following equation.
Small Vision System User Manual 3.4 39 Area Correlation Window Stereo analysis is the process of measuring range to an object based on a comparison of the object projection on two or more images. The fundamental problem in stereo analysis is finding corresponding elements between the images. Once the match is made, the range to the object can be computed using the image geometry. Area correlation compares small patches, or windows, among images using correlation.
Small Vision System User Manual Figure 3-7 Effects of the area correlation window size. At top is the original left intensity image. The greenscale images show windows of 7x7, 9x9, 11x11, and 13x13 windows (clockwise from upper left).
Small Vision System User Manual 3.5 41 Multiscale Disparity Multiscale processing can increase the amount of information available in the disparity image, at a nominal cost in processing time. In multiscale processing, the disparity calculation is carried out at the original resolution, and also on images reduced by 1/2. The extra disparity information is used to fill in dropouts in the original disparity calculation (Figure 3-8). Figure 3-8 Effects of multiscale disparity calculation.
Small Vision System User Manual 3.6 Filtering Like most vision algorithms, the results of stereo processing can contain errors. In the case of stereo, these errors result from noisy video signals, and from the difficulty of matching untextured or regularly textured image areas. Figure 3-9 shows a typical disparity image produced by the SRI algorithm. Higher disparities (closer objects) are indicated by brighter green (or white, if this paper is printed without color).
Small Vision System User Manual 43 left/right check can be implemented efficiently by storing enough information when doing the original disparity correlation. In practice, the combination of an interest operator and left/right check has proven to be the most effective at eliminating bad matches.
Small Vision System User Manual 3.7 Performance Using standard PC hardware, running either MS Windows 95/98/ME/2000/XP/NT or Linux, the SVS can compute stereo range in real time. Table 3-1 gives some typical timings for a 500 MHz Pentium III processor. Because the Stereo Engine has a very small memory footprint, the timings scale almost linearly with increasing processor speed.
Small Vision System User Manual 4 45 Calibration NOTE: There is a Calibration Addendum manual that details the exact steps necessary to perform calibration, and includes troubleshooting information. Please consult that manual for more detailed information about the calibration procedure. Most stereo camera setups differ from an ideal setup in which the cameras are perfect pinhole imagers and are aligned precisely parallel.
Small Vision System User Manual 4.1 Calibration Procedure An automatic calibration procedure using five image pairs of a planar calibration target is included as part of the smallv program. Given the image pairs of the calibration object, the system automatically locates corner features in the target, fits a model of the target to the images, and finally produces an estimate for the left and right camera intrinsics, the stereo head extrinsics, and the rectification matrices P0 , P1 , H 0 , and H1 .
Small Vision System User Manual Figure 4-4 Smallv calibration dialog window. 6. Detect the corner features in all views by pressing the features button in the lower bar of buttons. This cycles through all the images, displaying the corners in green as they are detected. If the feature finder fails on an image, please re-capture the image and redetect the features. When redetecting features for a single stereo pair, use the features button in the stereo pair box. 7.
Small Vision System User Manual 4.1.2 48 Calibration Target The standard calibration target, check.pdf, can be printed out on a single piece of 8.5 x 11 inch paper. In some cases, this image is too small, e.g., when using wide baseline cameras. There is another target, check2.pdf, that has squares twice as large as the standard one (54 mm instead of 27 mm). This target must be assembled from four printed pieces of 8.5 x 11 inch paper.
Small Vision System User Manual 5 49 API Reference – C++ Language With SVS 2.2x, the standard programming interface to the SVS libraries is in C++. To add stereo processing to your own programs, you call functions in the Stereo Engine library. These functions are available in svs.dll (Windows 95/98/2000/NT) or libsv.so (Unix systems). The header file is src/svsclass.h. The current version of the library is 2.2d. Source code samples for the C++ API are in the directory samples/.
Small Vision System User Manual 5.1 5.1.1 50 Threading and Multiple Stereo Devices Threading Issues The SVS core library functions (svs.dll, libsvs.so) are thread-safe: they can be used in any thread in a process. Of course, the user is responsible for not overlapping calls in different threads, e.g., starting up two competing disparity calculations using the same object in different threads. The MEGA-D and Dual-DCAM acquisition libraries are also thread-safe, in general.
Small Vision System User Manual 5.2 C++ Classes There are three main classes for SVS: classes that encapsulate stereo images, classes that produce the images from video or file sources, and classes that operate on stereo images to create disparity and 3D images. These classes are displayed in Figure 5-1. The header file is src/svsclass.h.
Small Vision System User Manual 52 // Make a video source object, using the loaded framegrabber interface svsVideoImages *videoObject = getVid eoObject(); // Make a stereo processing object svsStereoProcess *processObject = new svsStereoProcess(); // Open the video source bool ret = videoObject ->Open(); if (!ret) { …error code… } // Read in rectification parameters videoObject->ReadParams(“../data/megad-75.
Small Vision System User Manual 5.3 Parameter Classes svsImageParams svsRectParams svsDispParams Image frame size and subwindow parameters Image rectification parameters Image stereo processing (disparity) parameters Parameter classes contain information about the format or processing characteristics of stereo image objects. Each stereo image object contains an instance of each of the above classes.
Small Vision System User Manual 5.4 Stereo Image Class svsStereoImage Stereo image class The stereo image class encapsulates information and data for a single stereo image pair, along with any of its processed results, e.g., disparity image or 3D point cloud. Stereo image objects are usually produced by one of the image acquisition classes (svsVideoImages or svsFileImages), then processed further by an svsStereoProcess object. An svsStereoImage object holds information about its own state.
Small Vision System User Manual 5.4.3 Rectification Information bool isRectified; // have we done the rectification already? bool haveRect; // true if the rectification params exist svsRectParams rp; // rectification params, if they exist The images contained in a stereo image object (left, right and left-color) can be rectified, that is, corrected for intra-image (lens distortions) and inter-image (spatial offset) imperfections.
Small Vision System User Manual Each point is represented by a coordinate (X,Y,Z) in a frame centered on the left camera focal point. The Z dimension is distance from the point perpendicular to the camera plane, and is always positive for valid disparity values. The X axis is horizontal and the positive direction is to the right of the center of the image; the Y axis is vertical and the positive direction is down relative to the center of the image (a right-handed coordinate system).
Small Vision System User Manual 57 NULL, then no image data is inserted for that image. If the copy argument is true, then the buffer contents are copied onto the stereo image object’s own buffers. If not, then the input buffers are used internally.
Small Vision System User Manual 5.5 Acquisition Classes svsAcquireImages svsVideoImages svsFileImages SvsStoredImages Base class for all acquisition Acquire from a video source Acquire from a file source Acquire from a memory sourcde Acquisition classes are used to get stereo image data from video or file sources, and put it into svsStereoImage structures for further processing. During acquisition, images can be rectified, that is, put into a standard form with distortions removed.
Small Vision System User Manual 59 An acquisition object acquires stereo images and returns them when requested. These functions control the image streaming process. CheckParams() determines if the current acquisition parameters are consistent, and returns true if so. This function is used in video acquisition, to determine if the video device supports the modes that have been set. If the device is not opened, CheckParams() returns false. Start() starts the acquisition streaming process.
Small Vision System User Manual 5.6 60 Video Acquisition The video acquisition classes are subclasses of svsAcquireImages. The general class is svsVideoImages, which is referenced by user programs. This class adds parameters and functions that are particular to controlling a video device, e.g., frame size, color mode, exposure, and so on. Particular types of framegrabbers and stereo heads have their own subclasses of svsVideoImages.
Small Vision System User Manual Alternatively, a number can be used, giving the device in the order returned by the Enumerate() function, i.e., 1is the first device, 2 is the second, and so on. A value of 0 indicates any available device. Upon opening, the device characteristics are set to default values. To set values from a parameter file, use the ReadParams() function. A stereo device is closed and released by the Close() call. 5.6.
Small Vision System User Manual 62 The video frame parameters can be set independently, and not all combinations of values are legal. The CheckParams() function returns true if the current parameters are consistent. None of the frame or sampling mode parameters can be changed while images are being acquired, except for the offset parameters. These can be changed at any time, to pan and tilt the subwindow during acquisition. 5.6.
Small Vision System User Manual 5.7 63 File and Memory Acquisition The file and memory acquisition classes are subclasses of svsAcquireImages. This classes are used to input stereo images from files, or from arrays in memory, and present them for processing. Users who have their own stereo devices, and acquire images into memory, can use these classes to perform stereo processing with the SVS libraries.
Small Vision System User Manual 64 User images can also be copied into the object’s buffers, if copy is set to true. Here, the object manages all buffers, and the user program should not destroy them. Typically, a use program will have images stored in memory, and will want to rectify them as part of the stereo process. To do so, first load a parameter file into the acquisition object, using ReadParamFile. Next, turn on rectification by calling DoRect(true).
Small Vision System User Manual 5.8 Stereo Processing Classses svsStereoProcess svsMultiProcess Stereo processing class Multiscale stereo processing class The stereo processing classses perform stereo processing on stereo images encapsulated in an svsStereoImage object. The results are stored in the stereo image object. All relevant parameters, such as calibration information and stereo parameters, are also part of the stereo image object.
Small Vision System User Manual 5.9 Window Drawing Classes svsWindow svsDebugWin Window class for drawing 2D images Window class for printing output The window drawing classes output 2D stereo imagery to the display. The display window relies on the FLTK cross-platform windowing system (www.fltk.org), and provides basic graphical object drawing in addition to image display. The svsDebugWin class is for text output, useful when debugging programs.
Small Vision System User Manual These functions draw overlay information on the image displayed by svsWindow. There are two ways to draw overlays. One is to subclass svsWindow, overriding the DrawOverlay() function. Then, the subclass can perform any FLTK drawing within the subclass function. Another way to use overlays is to assign an overlay function to the svsWindow object, with DrawOverlayFn. This overlay function is called every time the overlay needs to be drawn.
