There are two classes of things whose motion has to be considered: the camera and the objects in the scene. One or both of these can be moving at any given time. We will first examine only object motion, leaving the incorporation of camera motion for later, more sophisticated techniques.
1 if |F(x,y,j) - F(x,y,k)| > t
This results in a binary difference picture, indicating which pixels have changed in intensity more than a given threshold, t , between frames j and k.
However, noise becomes a problem with video images -- individual pixels may change without an actual change to the objects in the image. There are two ways to make the difference detection process more robust. One is to make "super-pixels" by averaging larger squares of pixels before calculating the intensity change. However, the resulting image is grainy. The other method is to use a pixel mask and examine the box of pixels around each individual one to calculate the difference values.
All of the velocity vectors of the images in the scene will meet at the F.O.E. if the object is stationary. Moving objects will have a different direction of image flow. The depth, z, of a point can be determined from its horizontal displacement in space and the velocity of the observer.
In the sample below, the white trapezoid represents the part of the road that RALPH considered in making his steering decisions. In RALPH's case, the position and velocity of the camera are constant, simplifying that process. The program used was robust enough to steer the car if any parallel markings were visible on the road. Lane markers were the obvious choice for such markers, but RALPH was able to adapt to oil stains, sides of the road, and even trucks far ahead to aid in steering.
The process was simple: the pixels in the white trapezoid were converted into an overhead image, indicating the curve of the road ahead. To determine curvature, RALPH generated several test images, distorting the top lines of the image for different levels of turning.
By then summing up the intensity of the pixels of each column in a graph, the program was able to measure the image that had the sharpest discontinuities, and thus the straightest vertical features, and then calculate a turn based on that information.
RALPH also compares the current scanline profile with its current template, and is able to calculate the lateral shift necessary to stay in the middle of the lane. The template can be set either from RALPH's library, manually set by the driver, or dynamically determined based on the top rows of the trapezoid.