The projection of light rays onto the retina of the eye forms a two-dimensional image, but through combining the stereoscopic aspect of vision with other optical clues by means of some remarkably effective image-processing procedures, the viewer is able to perceive three-dimensional representations of scenes.

From Images to Surfaces proposes and examines a specific image-processing procedure to account for this remarkable effect-a computational approach that provides a framework for understanding the transformation of a set of images into a representation of the shapes of surfaces visible in a scene. Although much of the analysis is applicable to any visual information processing system-biological or artificial-Grimson constrains his final choice of computational algorithms to those that are biologically feasible and consistent with what is known about the human visual system.

In order to clarify the analysis, the approach distinguishes three independent levels: the computational theory itself, the algorithms employed, and the underlying implementation of the computation, in this case through the human neural mechanisms. This separation into levels facilitates the generation of specific models from general concepts.

This research effort had its origin in a theory of human stereo vision recently developed by David Marr and Tomaso Poggio. Grimson presents a computer implementation of this theory that serves to test its adequacy and provide feedback for the identification of unsuspected problems embedded in it. The author then proceeds to apply and extend the theory in his analysis of surface interpolation through the computational methodology.

This methodology allows the activity of the human early visual system to be followed through several stages: the Primal Sketch, in which intensity changes at isolated points on a surface are noted; the Raw 2.5-D Sketch, in which surface values at these points are computed; and the Full 2.5-D Sketch, in which these values—including stereo and motion perception—are interpolated over the entire surface. These stages lead to the final 3-D Model, in which the three-dimensional shapes of objects, in object-centered coordinates, are made explicit.