In recent years, numerous prototypical systems have been developed for multisensor data fusion. A typical data fusion process operates on sensor parametric data (e.g., data related to target position or attribute data) in order to develop an order of battle, provide an evaluation of tactical situations, or assess tactical threats. This model, developed by the Data Fusion Sub- panel (DFS) of the Joint Directors of Laboratories, partitions fusion processing into four conceptual levels. Ancillary functions in a fusion system include the human computer interface, data base management, source-preprocessing functions, and communications. Military applications for data fusion span a broad range including fusion of data on board a single platform for identifying other platforms (e.g., identification--friend or foe--neutral systems), threat warning systems, situation assessment, and threat assessment systems. Large scale systems such as the All-Source Analysis System (ASAS) or the Joint Surveillance, Targeting, and Reconnaissance System (JSTARS) provide for direction, coordination, and fusion of both ground-based and airborne sensors to aid in the effective management of a ground based battlefield environment. Such systems have become ever more sophisticated. Indeed, many of the prototypical systems utilize advanced identification techniques such as knowledge-based or expert systems. Dempster-Shafer interface techniques, adaptive neural networks, and sophisticated tracking algorithms. While much research is being performed to develop and apply new algorithms and techniques, little work has been performed to determine how well such methods work or to compare alternative methods against a common problem. The issues of system performance and system effectiveness are keys to establishing how well an algorithm, technique, or collection of techniques perform, and then the extent to which these techniques may be used to achieve success on an operational mission.

The successful design and operation of autonomous or partially autonomous vehicles which are capable of traversing uncertain terrains requires the application of multiple sensors for tasks such as: local navigation, terrain evaluation, and feature recognition. In applications which include a teleoperation mode, there remains a serious need for local data reduction and decision-making to avoid the costly or impractical transmission of vast quantities of sensory data to a remote operator. There are several reasons to include multi-sensor fusion in a system design: (i) it allows the designer to combine intrinsically dissimilar data from several sensors to infer some property or properties of the environment, which no single sensor could otherwise obtain; and (ii) it allows the system designer to build a robust system by using partially redundant sources of noisy or otherwise uncertain information. At present, the epistemology of multi-sensor fusion is incomplete. Basic research topics include the following taskrelated issues: (i) the value of a sensor suite; (ii) the layout, positioning, and control of sensors (as agents); (iii) the marginal value of sensor information; the value of sensing-time versus some measure of error reduction, e.g., statistical efficiency; (iv) the role of sensor models, as well as a priori models of the environment; and (v) the calculus or calculi by which consistent sensor data are determined and combined. In our research on multi-sensor fusion, we have focused our attention on several of these issues. Specifically, we have studied the theory and application of robust fixed-size confidence intervals as a methodology for robust multi-sensor fusion. This work has been delineated and summarized in Kamberova and Mintz (1990) and McKendall and Mintz (1990a, 1990b). As we noted, this previous research focused on confidence intervals as opposed to the more general paradigm of confidence sets. The basic distinction here is between fusing data characterized by an uncertain scalar parameter versus fusing data characterized by an uncertain vector parameter, of known dimension. While the confidence set paradigm is more widely applicable, we initially chose to address the confidence interval paradigm, since we were simultaneously interested in addressing the issues of: (i) robustness to nonparametric uncertainty in the sampling distribution; and (ii) decision procedures for small sample sizes. Recently, we have begun to investigate the multivariate (confidence set) paradigm. The delineation of optimal confidence sets with fixed geometry is a very challenging problem when: (i) the a priori knowledge of the uncertain parameter vector is not modeled by a Cartesian product of intervals (a hyper-rectangle); and/or (ii) the noise components in the multivariate observations are not statistically independent. Although it may be difficult to obtain optimal fixed-geometry confidence sets, we have obtained some very promising approximation techniques. These approximation techniques provide: (i) statistically efficient fixed-size hyper-rectangular confidence sets for decision models with hyper-ellipsoidal parameter sets; and (ii) tight upper and lower bounds to the optimal confidence coefficients in the presence of both Gaussian and non-Gaussian sampling distributions. In both the univariate and multivariate paradigms, it is assumed that the a priori uncertainty in the parameter value can be delineated by a fixed set in an n-dimensional Euclidean space. It is further assumed, that while the sampling distribution is uncertain, the uncertainty class description for this distribution can be delineated by a given class of neighborhoods in the space of all n-dimensional probability distributions. The following sections of this paper: (i) present a paradigm for multi-sensor fusion based on position data; (ii) introduce statistical and set-valued models for sensor errors and a priori environmental uncertainty; (iii) explain the role of confidence sets in statistical decision theory and sensor fusion; (iv) relate fixed-size confidence intervals to fixedgeometry confidence sets; and (v) examine the performance of fixed-size hyper-cubic confidence sets for decision models with spherical parameter sets in the presence of both Gaussian and non-Gaussian sampling distributions

Two different types of adaptive networks are considered for solving the centralized and distributed hypothesis testing problem. The performance of the two different types of networks is compared under different performance indices and training rules. It is shown that training rules based on the Neyman-Pearson criterion outperform error based training rules. Simulations are provided for data that are linearly and nonlinearly separable.

An intuitive architecture for neural net multisensor data fusion consists of a set of independent sensor neural nets, one for each sensor, coupled to a fusion net. Each sensor is trained from a representative data set of the particular sensor to map to an hypothesis space output. The decision outputs from the sensor nets are used to train the fusion net to an overall decision. In this paper the sensor fusion architecture is applied to an experiment involving the multisensor observation of object deployments during the recent Firefly launches. The deployments were measured simultaneously by X-band, CO2 laser, and L-band radars. The range-Doppler images from the X-band and CO2 laser radars were combined with a passive IR spectral simulation of the deployment to form the data inputs to the neural sensor fusion system. The network was trained to distinguish predeployment, deployment, and postdeployment phases of the launch based on the fusion of these sensors. The success of the system is utilizing sensor synergism for an enhanced deployment detection is clearly demonstrated.

Multi-sensor fusion is widely used in object recognition and classification at present, since this technique can efficiently improve the accuracy and the ability of fault tolerancy. This current presentation described a sub-system of multi-sensor integration: range and intensity image fusion system, which is model-based and used for object recognition and classification. In the data fusion process, the Dempster-Shafer's Evidential Reasoning (DSER) is selected and used for the data fusion at report level. This presentation mainly discussed the construction of the Basic Probability Assignments Function in DSER for the recognition and classification, and the decision strategies based on the fused information. At the same time, by comparing the experimental results based on separate original data and the fused data respectively, it is shown that the latter is more accurate than the first ones.

The application of numeric methods to the minimization of error has become an emerging paradigm for obj ect recovery. Typically, a parametric representation describing the object is postulated. Its parameters are then adjusted to minimize some measurement of the distance between the representation and the datapoints (the error-of-fit model). Characteristics of the sensor used to recover the points may be implicit in this formulation or may not be included at all. While sensors may be precise for a specific field of view no sensor is everywhere exact. A laser range finder for example, yields very sharp x- and y-coordinate values; however, its z-coordinate is less trustworthy. It becomes important to capture the strengths and weaknesses of a sensor and incorporate them into the recovery process. We seek to make explicit the contribution of a particular sensor by introducing a sensor model. This partitioning facilitates the development of an appropriate description of a sensor's characteristics. Also, it helps clarify interactions among different aspects of the recovery process ( i.e. error-of-fit model, sensor model, and parametric object representation). The sensor model is reflected in the certainty of sensed quantities (position, color, intensity) associated with a datapoint. We explore whether the introduction of an explicit sensor model yields an improvement in the recovery process. The PROVER (Parametric Representation Of Volumes: Experimental Recovery) System, a testbed used in the development of sensor models is described.

This paper presents an approach to sensor-based decision making in unstructured environments that relies on describing geometric structures by parameterized volumes. This approach leads to large systems of nonlinear, stochastic inequalities. We describe how these inequalities can be solved using interval bisection, discuss the structural and statistical convergence of the technique, and present some preliminary experimental results.

A method for reconstruction of 3D object models from multiple views of range image is proposed. It is very important to use these partially redundant data effectively to get an integrated, complete and accurate object model. The object shape is unconstrained, curved surfaces are allowed. From each view of range image, surfaces are segmented and fitted into planar and quadratic patches by a robust residual analysis method (we address this method in another paper). Analyzing the errors of fitted surfaces from each view, the final expressions of the surfaces are merged from every view. A boundary representative model (B-rep) is used to express the final complete object. The method can be used to create 3D models for object recognition.

In many situations, only some parts of an object are visible while other parts are occluded. In other situations, information about an object is available piecemeal as the parts are scanned sequentially, such as when eye-motions are used to explore an object. Part information is also crucially important for objects with articulating parts, or with removable parts. In all of these cases, the sensor-scanner system must divide an object into subcomponents, and must also be able to integrate the part-information using appropriate data concerning the spatial relationships among the parts as well as the temporal scan sequences. This work describes how such issues are addressed in recognizing human faces from their parts using a neural network approach. Parallels are drawn between neurophysiological and psychophysical experiments, as well as deficits in visual object recognition. This work extends our existing modular system, developed for learning and recognizing 3D objects from multiple views, by investigating the capabilities which need to be augmented for coping with objects which are represented hierarchically. The ability of the previous system to learn and recognize 3D objects invariant to their apparent size, orientation, position, perspective projection, and 3D pose serves as a strong foundation for the extension to more complex 3D objects.

A class of image sequences can be characterized as being spatially invariant and linearly additive based on their image formation processes. In these kinds of sequences, all features are positionally invariant in each image of a given sequence but have varying gray-scale properties. The various features of the scene contribute additively to each image of the sequence but the image-formation processes associated with given features have characteristic signatures describing the manner in which they vary over the image sequence. Examples of appropriate image sequences include multispectral image sequences, certain temporal image sequences, and NMR image sequences generated by modification of the excitation parameters. Note that image sequences can be formed using a variety of imaging modalities as long as the linearly additive and spatially invariant requirements are not violated. Features associated with different image-formation processes generally will have unique signatures that can be used to generate linear filters for isolating selected image-formation processes or for performing data compression. Starting with an explicit mathematical model, techniques are presented for generating optimal filters using simultaneous diagonalization for enhancement of desired image-formation processes and data compression with this class of image sequences. A unique property of this approach is that even if several image-formation processes occupy a given pixel, they can still be isolated.

One of the main problems in constructing a supervisory robot seems to be the excess of available real-time information. The trend is to make a robot obtaining data from a lot of sensors before it takes a decision. Multiplicity of sensors increases hardware costs and control software complexity. A robot usually needs only a small part of all available information for an efficient performance. The current work will demonstrate a new approach to design of information-efficient supervisory robots. The theory combines the basic concepts of discrete event control extended to stochastic systems with some ideas of information economics. A controlled system behavior is modeled as a sequence of uncontrolled events and robot's decisions. Decisions are based on events observed by robot, and each event needs a sensor to be observed. The model provides a formal statement of the sensors selection problem. A solution procedure of the stated problem will be demonstrated, and some applications of the method to different types of robotic control will be presented.

By virtue of their low cost and simplicity, ultrasonic sensors are widely used in rangefinding applications. However, when a more detailed map of the environment is needed, as is the case in mobile robotics, the simple-minded use of a single ultrasonic sensor is often insufficient. In this paper we propose a measurement system composed of three ultrasonic sensors, one transmitting and three receiving, to overcome these problems. By combining the information from the three sensors we can accurately determine the position, both in distance and bearing, of all objects in the field of view taking one single snapshot of the scene. Within the limits derived in this paper, it is even possible to discriminate between different types of reflectors. To get these results we have to find the reflections belonging to the same object in the signals received by the three sensors, a problem analogous to the correspondence problem in stereo- vision. We present a maximum likelihood approach as a solution to this problem. A realistic scene will be used to compare our sensor with the traditional ultrasonic ranging sensor.

We summarize a recently developed modular neural system which exploits sequences of 2D views for learning and recognizing 3D objects. An aspect network is an unsupervised module of our complete artificial vision system for detecting and learning the view transitions (as the appearance of a rotating object changes), and for later recognizing objects from sequences of views. By processing sequences of views, the system accumulates evidence over time, thereby increasing the confidence of its recognition decisions. Also, when new views are revealed following views recognized previously by an aspect network during the course of observation, the new views and view-transitions are used to refine the evolving 3D object representation automatically. Recognition is possible even from novel (previously unexperienced) view sequences. The objects used for illustration are model aircraft in flight. The computations are formulated as differential equations among analog nodes and synapses to model the temporal dynamics explicitly.

This paper is about the interface between continuous and discrete robot control. We advocate encapsulating continuous actions and their related sensing strategies into behaviors called situation specific activities, which can be constructed by a symbolic reactive planner. Task- specific, real-time perception is a fundamental part of these activities. While researchers have successfully used primitive touch and sonar sensors in such situations, it is more problematic to achieve reasonable performance with complex signals such as those from a video camera. Active vision routines are suggested as a means of incorporating visual data into real time control and as one mechanism for designating aspects of the world in an indexical-functional manner. Active vision routines are a particularly flexible sensing methodology because different routines extract different functional attributes from the world using the same sensor. In fact, there will often be different active vision routines for extracting the same functional attribute using different processing techniques. This allows an agent substantial leeway to instantiate its activities in different ways under different circumstances using different active vision routines. We demonstrate the utility of this architecture with an object tracking example. A control system is presented that can be reconfigured by a reactive planner to achieve different tasks. We show how this system allows us to build interchangeable tracking activities that use either color histogram or motion based active vision routines.

The purpose of sensing is to collect information about the environment, with which the system interacts, to assist in performing tasks. For the system to be efficient and reliable in accomplishing tasks, sensing should provide information which is key to the task accomplishment. The ability to automatically reconfigure sensors between operations to collect sensory data enables the planning of sensing strategies for achieving this goal. The sensory action of acquiring data, however, will improve knowledge about the environment; hence improved knowledge could be utilized in determining subsequent sensory actions suitable for the increasingly-understood environment. Thus, an adaptive sensing strategy is more desirable than a pre-determined plan. In this paper, we demonstrate how the techniques of Bayesian decision theory can be used to develop sensing strategies which are adaptive and goal-directed. We emphasize how to model undertainties of sensory outcomes to improve the robustness of task achievement. The methods were applied to the problem of identifying and localizing electrical components using a camera mounted on a robot arm. This implementation is described and the automatically-generated strategies are discussed.

Vision systems are a possible choice to obtain sensorial data about the world in robotic systems. To obtain three-dimensional information using vision we can use different computer vision techniques such as stereo, motion, or focus. In particular, this work explores focus to obtain depth or structure perception of the world. In practice, focusing can be obtained by displacing the sensor plate with respect to the image plane, by moving the lens, or by moving the object with respect to the optical system. Moving the lens or sensor plate with respect to each other causes changes of the magnification and corresponding changes on the object coordinates. In order to overcome these problems, we propose varying the degree of focusing by moving the camera with respect to the object position. In our case, the camera is attached to the tool of a manipulator in a hand-eye configuration, with the position of the camera always known. This approach ensures that the focused areas of the image are always subjected to the same magnification. To measure the focus quality we use operators to evaluate the quantity of high-frequency components on the image. Different types of these operators were tested and the results compared.

We propose an alternative way to study the problem of visual recognition which is closer to the spirit emerging from Brooks' work on building robots than to Marr's reconstructive approach. Our theory is purposive in the sense that recognition is considered in the context of an agent performing it in an environment, along with the agent's intentions that translate into a set of behaviors; it is qualitative in the sense that only partial recovery is needed; it is active in the sense that various partial recovery tasks need for recognition are achieved through active vision; and it is opportunistic in the sense that every available cue is used.

Fusion of kinematic measurements from spatially separated and/or independently moving sensors can be considerably simplified by judicious choice of coordinate systems. This paper discusses kinematics and issues associated with sensor-level (measurement) and central-level (fusion) coordinate systems. Compensation for sensor motion is briefly discussed and a coordinate system for angle-only tracking is discussed that avoids some of the difficulties associated with polar or spherical coordinates.

In this paper we present an overview of the MVP sensor planning and modeling system that we have developed. MVP automatically determines camera viewpoints and settings so that object features of interest are simultaneously visible, inside the field-of-view, in-focus and magnified as required. We have analytically characterized the domain of admissible camera locations, orientations and optical settings for each of the above feature detectability requirements. In addition, we have posed the problem in an optimization setting in order to determine viewpoints that simultaneously satisfy all previous requirements. The location, orientation and optical settings of the computer viewpoint are achieved in the employed sensor setup by using sensor calibration models. For this purpose, calibration techniques have been developed that determine the mapping between the parameters that are planned and the parameters that can be controlled in a sensor setup which consists of a camera in a hand-eye arrangement equipped with a lens that has zoom, focus and aperture control. Experimental results are shown of these techniques when all the above feature detectability constraints are included. In order to verify satisfaction of these constraints, camera views are taken from the computed viewpoints by a robot vision system that is positioned and its lens is set according to the results of this method.

In this paper, we discuss techniques for extending the sensor planning capabilities of the machine vision planning system to include motion in a well-known environment. In a typical work cell, vision sensors are needed to monitor a task and provide feedback to motion control programs or to assess task completion or failure. In planning sensor locations and parameters for such a work-cell, all motion in the environment must be taken into account in order to avoid occlusions of desired features by moving objects and, in the case where the features to be monitored are being manipulated by the robot, to insure that the features are always within the camera's view. Several different sensor locations (or a single, movable sensor) may be required in order to view the features of interest during the course of the task. The goal is to minimize the number of sensors (or to minimize the motion of the single sensor) while guaranteeing a robust view at all times during the task, where a robust view is one which is unobstructed, in focus, and sufficiently magnified. In the past, sensor planning techniques have primarily focused on static environments. We present techniques which we have been exploring to include knowledge of motion in the sensor planning problem. Possible directions for future research are also presented.

Our work in teleoperation highlights its space applications, and related tasks such as remote platform servicing, telescience, and lunar exploration. These tasks are complex, time- consuming, and relatively unstructured. Demands for manual dexterity are often high; the work is fatiguing; and uncertainty, which includes effects of time-delay, is nearly always present. In the face of these problems, we have been working along several technical fronts which include redundant telemanipulator control, multi-camera viewing and real-time graphics simulation, integrated operator interface design, and systems-scale ground laboratory experiments. Our main experimental thrust is end-to-end performance characterization--formal experiment design, task instrumentation for real-time data capture, integrated system demonstrations, and human factors analysis. Collectively, the goal is to quantify operator limitations, component technology requirements, and their interdependencies, all in the context of meaningful tasks with realistically posed system-level operational constraints (lighting, task geometry, time-delay, control & communication bandwidths, viewing & display limitations, etc.). Accompanying technical issues are reduction of operator error, workload, and training, each in itself a significant risk and cost driver for space operations. Toward these ends, we have been emphasizing advanced approaches to teleoperation, in the functional areas of task perception, planning, and control.

For reliable navigation, a mobile robot needs to be able to recognize where it is in the world. We describe an efficient and effective image-based representation of perceptual information for place recognition. Each place is associated with a set of stored image signatures, each a matrix of numbers derived by evaluating some measurement function over large blocks of pixels. Measurements are chosen to be characteristic of a location yet reasonably invariant over different viewing conditions. Signature matching can be done quickly by element wise comparison. Additional stability can be gotten by matching signatures at offsets or across scales. Signatures can be stored in a k-d tree so that retrieval of similar signatures is fast. We can also use several types of measurements in tandem to enhance recognition accuracy. We present preliminary experimental results which show up to 90% recognition accuracy. When used together with prior position information, we suggest that this performance is good enough to support reliable place recognition from a series of images.

The performance of autonomous mobile robots performing complex navigation tasks can be dramatically improved by directly expensive sensing and planning in service of the task. The task-direction algorithms can be quite simple. In this paper we describe a simple task-directed vision system which has been implemented on a real outdoor robot which navigates using stereo vision. While the performance of this particular robot was improved by task-directed vision, the performance of task-directed vision in general is influenced in complex ways by many factors. We briefly discuss some of these, and present some initial simulated results.

While in NASA's earlier space missions such as Voyager the number of sensors was in the hundreds, future platforms such as the Space Station Freedom will have tens of thousands sensors. For these planned missions it will be impossible to use the comprehensive monitoring strategy that was used in the past in which human operators monitored all sensors all the time. A selective monitoring strategy must be substituted for the current comprehensive strategy. This selective monitoring strategy uses computer tools to preprocess the incoming data and direct the operators' attention to the most critical parts of the physical system at any given time. There are several techniques that can be used to preprocess the incoming information. This paper presents an approach to using diagnostic reasoning techniques to preprocess the sensor data and detect which parts of the physical system require more attention because components have failed or are most likely to have failed. Given the sensor readings and a model of the physical system, a number of assertions are generated and expressed as Boolean equations. The resulting system of Boolean equations is solved symbolically. Using a priori probabilities of component failure and Bayes' rule, revised probabilities of failure can be computed. These will indicate what components have failed or are the most likely to have failed. This approach is suitable for systems that are well understood and for which the correctness of the assertions can be guaranteed. Also, the system must be such that assertions can be made from instantaneous measurements. And the system must be such that changes are slow enough to allow the computation.

Behavioral robotics is the specification and control of robots as a composition, coordination, and adaptation of more primitive sub-behaviors. The past decade has led to the development of many useful algorithms that can support valuable behavioral function in robots (e.g., perception, robotics, planning, domain rules). Increased availability of these component algorithms increases the need for robot control methods that can select between gross behaviors in addition to providing control within a given behavior. For example, gross behavioral changes may consider when and whether to attend to new stimuli, and if and how that stimuli can lead to new or different behaviors. This paper describes an approach being developed for the specification and control of these types of behavioral programs. The first section introduces a task oriented approach to behavioral robot program specification and control. The second section then describes a Behavioral Architecture for Robot Tasks (BART) being developed. A BART language is being built to provide a portable tool to support various robot programming and execution strategies, evidence accrual methods, and domain representations. This language is being used to develop behavioral programs that control tanks and tank platoons.

It is usually difficult to use underwater robots for mapping, reconnaissance, and mine-clearing tasks in shallow water (10 to 80 foot depth) ocean environments. The shallow water environment is characterized by strong, intermittent wave surge which requires robot behaviors that are capable of riding out the surge and then repositioning the platform and re- acquiring the objects being sensed. The shallow water area is also characterized by water that is murky, making optical sensors useless for long range search, and which produces multiple paths for sonar returns, giving errant range readings. Teleoperation from a remote surface platform is not effective due to the rapid changes in the environment. A more promising approach would place reactive intelligence on-board the robot. This paper describes such an approach which uses high frequency acoustic and vision sensing and a situated reasoning software architecture to provide task-achieving capability to an underwater robot in a shallow water environment. The approach is demonstrated in the context of a shallow water marking task wherein a robot must locate and navigate to a moored object in shallow water depths, attach a buoyant marker, and then return to a destination location. The approach seeks to integrate selective perception with robust transit and hovering behaviors to overcome the natural problems associated with shallow water environments.

We are interested in visual capabilities that enable robots to act in dynamic, unstructured domains. Towards this end, we have designed a vision system that has the capability of segmenting and tracking a moving object. Our system has been implemented and runs in real- time on a Connection Machine. We have configured a HERO-2000 robot to use the vision system for control. Based on object centroids provided by the vision system, the robot pans to follow a large moving object in the field of view. The robot has proven capable of reliably tracking a moving person for several minutes. In this paper, we present the visual capabilities we have constructed and justify their utility for unstructured environments. We then describe our algorithms, their implementation, and their use in the panning robot.

We describe a monocular 'eye in hand' approach to acquire 3-D information of a scene in a robotic environment. By tracking points through an image sequence taken from a moving camera, the correspondence and occlusion problem is solved. The problem of dynamically selecting points in real time is addressed. A method for defining a qualitative measure of the 'trackability' of points is introduced. To efficiently use the available multitarget tracking hardware a scheme of prediction and workload balancing is shown. To overcome the inaccurate knowledge of the exterior orientation of the camera (which is mounted on a robot) control points are placed in the scene and, by means of resection, the exact exterior orientation is determined. Positional information is extracted from the 3-D data enabling the robot to grab the object it monitored.

This paper presents a method for the segmentation of multiple motions in a scene using the singular value decomposition of a feature track matrix. It is shown that motions can be separated using the right singular vectors associated with the nonzero singular values. This is based on the relationship between the right singular vectors and the principal components of the covariance matrix of the tracks. Furthermore, under general assumptions, the number of numerically nonzero singular values can be used to determine the number of motions. This can be used to derive a relationship between a good segmentation, the number of nonzero singular values in the input and the sum of the number of nonzero singular values in the segments. The approach is demonstrated on real and synthetic examples and a study of the robustness of the method is given. The paper ends with a critical analysis of the approach.

In this paper, a new formulation and method is presented to directly recover 3D short term motion from range image sequences. In the case of a rigid-body motion, the formulation relates through a set of linear equations the six motion parameters to the first spatial-temporal derivatives and coordinates of a point. A weighted least square method is used to find the solution of this equation set. In case of locally rigid motion, the six rigid motion parameters of each point are estimated from the first and second spatial-temporal derivatives. For each point, a set of 10 linear equations with six unknowns is again solved by the least square method. The special case of local translation with small rotation gives a very elegant closed-form solution and an explicit geometric explanation. We also shown that the formulation can be easily generalized to any arbitrary motion. The proposed formulation has theoretical elegance since it only involves solving a set of linear equations. Results on both synthetic and real data are given.

The brain represents perceptual and motor information in several reference frames (for example body-centered, object-centered, or retinal-centered reference frames). In a simple sensory-motor program such as looking at and taking an object, at least three fundamental processes must be carried out by the cerebral cortex; (1) in order to recognize the target object, the cortex has to transform the pattern of excitation on the retina from a retinotopic coordinate system to a coordinate system centered on the object itself; (2) in order to bring a hand to the desired position in space, the cortex must transform the visual information related to the target location (relative to the hand) into an appropriate motor command of the reaching hand; (3) in order to guide coherent behavioral actions, more complex sensory-motor programs (for example, conditional reaching of a target) are constructed from time-dependent relations between these basic transformations. The cortex correlates sensory and motor events and learns to prepare responses to forthcoming events. Neurophysiological data on the motor area of the monkey allowed us to model the coordinate transformations from body-centered to arm-centered reference frames involved in the command of arm reaching movements in 3-D space. Anatomical and neuropsychological data suggest similar coordinate transformations along the visual pathway to relate retinal-centered to object-centered reference frames and we have thus extended the model to this coordinate transformation. Time integration seems to proceed differently since internal representations of programs are dynamically constructed. Available physiological and anatomical data on frontal areas (and particularly prefrontal cortex) help to predict specific learning mechanisms for time processing and then construct a model for learning sensory-motor sequences.

The brain is a complex perceptual-motor system. To assemble an understanding of this system we need some ploy to get a grip on this complexity. Beginning at the level of individual muscles, a functional model for organizing the brain is developed. This forms the basis for an approach to the design of intelligent robotic systems based on the idea of composing complex systems from primitive functional modules. This paper addresses the sensor fusion aspects of this composition. We identify perceptual components of primitive functional modules, and address the problems of sensor fusion when composing complex perceptual systems employing an array of environmental sensory modalities. The discussion is centered on the human perceptual-motor system.

A network is described that can be used for multiple targets grouping and tracking or directing a vision system's focus of attention. The network models a biologically plausible astroglial- neural network in the visual cortex whose parameters are tuned to match a psychophysical database on apparent motion. The architecture consists of a diffusion layer and a contrast- enhancement layer coupled by feedforward and feedback connections; input is provided by a separate feature extracting layer. The dynamics of the diffusion-enhancement bilayer exhibit grouping of static features on multiple scales as a function of time, and long-range apparent motion between time varying inputs. The model is cast as a parallel analog circuit which is realizable in VLSI. We present simulations that reproduce static grouping phenomena useful for multiple target grouping and tracking over multiple scales, demonstrate several long-range apparent motion phenomena, and discuss single targets that split, and multiple targets that merge.

The high resolution field of view of the human eye only covers a tiny fraction of the total field of view. While this arrangement allows for great economy in computational resources, it forces the visual system to solve others that would not exist with a wide field of uniform high resolution. One of these problems is how to determine where to redirect the fovea given only the low-resolution information available in the periphery. The advent of spatially-variant receptor arrays for cameras has made it imperative that computational solutions to this problem be found. While the use of motion in this role is well accepted, color has been associated strongly with foveal vision. We show that color cues are well preserved under low resolution and illustrate an algorithm for locating objects based on color histograms that is both effective under low resolution and computationally efficient.

A decentralized Bayesian hypothesis testing problem is considered. It is analytically demonstrated that for the binary hypothesis problem, when there are two sensors with statistically independent Gaussian-distributed observations (conditioned on the true hypothesis), there is no loss in optimality in using the same decision rule at both sensors. Also, a multiple hypothesis problem is considered; some structure is analytically established for a optimal set of decision rules.

The asymptotic performance of binary- and multilevel-logic distributed decision fusion systems is studied. We define as perfect detectability the condition at which the detection probability at the fusion approaches one as the number of available decisions at the fusion approaches infinity, while the false alarm probability approaches zero. We investigate the performance of the fusion when the sensors transmit to the fusion the output of binary or multi-level memoryless nonlinearities. When the binary or multi-level decisions that the sensors transmit are independent conditioned on either hypothesis, it is shown that, under a Neyman-Pearson fusion rule, perfect detectability is achievable at an exponential rate for similar and dissimilar sensors, provided that the receiver operating characteristic (ROC) of each sensor lies above a lower bound ROC. The lower bound ROC has been calculated numerically for the case of similar sensors. The resulting lower bound ROC is very 'liberal,' in the sense that the ROC of any reasonable detector lies above it.

To achieve continuous operation and thus facilitate use of vision in a dynamic scenario, it is necessary to introduce a purpose for the visual processing. This provides information that may control the visual processing and thus limits the amount of resources needed to obtain the required results. A proposed architecture for vision systems is presented, along with an architecture for visual modules. This architecture enables both goal and data driven processing, with a potentially changing balance between the two modes. To illustrate the potential of the proposed architecture, a sample system for recovery of scene depth is presented, with experimental results which demonstrate a scalable performance.

In this sequel to work previously completed by Korolov and Chen, we design the end-point position control of a one-link flexible robot arm subjected to a wise spectrum of operational conditions. The linear control scheme designed here treats the possible variations of natural frequencies as uncertainties and is robust if the uncertainties stay inside specified bounds. The results of the simulation of this scheme are compared with simulations of the non-linear controller for the same arm.

DIGNET is a self-organizing artificial neural network (ANN) that exhibits deterministically reliable behavior to noise interference, when the noise does not exceed a pre-specified level of tolerance. The complexity of the proposed ANN, in terms of neuron requirements versus stored patterns, increases linearly with the number of stored patterns and their dimensionality. The self-organization of the DIGNET is based on the idea of competitive generation and elimination of attraction wells in the pattern space. DIGNET is used for pattern recognition and classification. Analytical and numerical results are included.

A biologically-inspired, behavior-based architecture for the integration of sensor data used by robotics systems is introduced, and examples are given which illustrate the control of a simple mobile robot, and also a ball-catching robot. The paper also goes on to discuss the extension of the concepts for more general robotic systems. The limitations of the subsumption architecture for general robotic use are explored in the context of mobile robots.

The European Community's strategic research initiative in information technology (ESPRIT) has been in place for nearly five years. An early example of the pan-European collaborative projects being conducted under this initiative is 'SKIDS': Signal and Knowledge Integration with Decisional Control for Multisensory Systems. This four year project, which is approaching completion, aims to build a real-time multisensor perception machine. This machine will be capable of performing data fusion, interpretation, situation assessment, and resource allocation tasks, under the constraints of both time and resource availability, and in the presence of uncertain data. Of the many possible applications, the surveillance and monitoring of a semi-automated 'factory environment' has been chosen as a challenging and representative test scenario. This paper presents an overview of the goals and objectives of the project, the makeup of the consortium, and roles of the members within it, and the main technical achievements to data. In particular, the following are discussed: relevant application domains, and the generic requirements that can be inferred from them; sensor configuration, including choice, placement, etc.; control paradigms, including the possible trade-offs between centralized, hierarchical, and decentralized approaches; the corresponding hardware architectural choices, including the need for parallel processing; and the appropriate software architecture and infra-structure required to support the chosen task oriented approach. Specific attention is paid to the functional decomposition of the system and how the requirements for control impact the organization of the identified interpretation tasks. Future work and outstanding problems are considered in some concluding remarks. By virtue of limited space, this paper is descriptive rather than explanatory.

This paper presents the paradigm of interactive and cooperative sensing and control as a fundamental mechanism of integrating and fusing the strengths of man and machine for advanced teleoperation. The interactive and cooperative sensing and control is considered as an extended and generalized form of traded and shared control. The emphasis of interactive and cooperative sensing and control is given to the distribution of mutually nonexclusive subtasks to man and machine, the interactive invocation of subtasks under the man/machine symbiotic relationship, and the fusion of information and decision-making between man and machine according to their confidence measures. The proposed interactive and cooperative sensing and control system is composed of such major functional blocks as the logical sensor system, the sensor-based local autonomy, the virtual environment formation, and the cooperative decision-making between man and machine. A case study is performed to demonstrate the feasibility of implementing the fundamental theory and system architecture of interactive and cooperative sensing and control, proposed for the new generation of teleoperation.

This paper presents algorithms to create a robust CAD-compatible surface model of an object utilizing data input from an active ranging device. The algorithm is divided into two stages, each of which approximates the data. The first stage 'cleans' the data and creates a dense grid. The second stage uses uniform B-Splines as the finite element to create a global description of the surface. This technique is a regularized approximation and can deal with data which does not necessarily have to lie on a grid, or be regularly spaced. The first stage of the algorithm eliminates outliers, while the second stage smooths over Gaussian noise. The algorithm can also be used to reconstruct a surface from sparse binocular stereo disparities including errors due to mismatches. The algorithms have been implemented and tested on a wide variety of data: sparse data from binocular stereo, depth measurements from laser radar, and depth measurements from active triangulation using a plane of light. In this paper, we present results of varying the parameters of the second stage of the model, in order to better understand the behavior of the algorithm, and to provide users with quantitative measures of the effect of the variations in order to choose the optimal parameters for a particular application.

Two related and important problems in the field of model-based computer vision are the extraction of predefined primitives from geometric data, and the computation of correspondences among such primitives. We show that both problems are equivalent to the optimization of a cost function, which often has very many local minima. One implication of this model is that a robust algorithm for these problems must find the global minimum of the associated cost function from among the local minima. The minimal subset principle states that a small subset of a set is often able to encode the characteristics of the entire set. For primitive extraction a minimal subset is the smallest number of points necessary to define a geometric primitive. Similarly, for correspondence computation a minimal subset is the smallest number of correspondences between geometric and model primitives necessary to define a pose (position and orientation) of the model. Randomly choosing such minimal subsets and evaluating them by using a cost function is a general and robust way to perform primitive extraction and correspondence computation. The main difficulty with this approach is that sometimes a large number of random samples (and therefore cost function evaluations) are necessary. We use a genetic algorithm to decrease the number of random samples significantly, and therefore to decrease the execution time. Some approaches to speeding up minimal subsets using algorithms on different parallel architectures are also described.

The problem of surface reconstruction from sparse visual depth data is studied. Because of the difficulties caused by outliers in the depth data and by overlapping data points from multiple surfaces, the reconstruction problem is posed as a data clustering problem. This problem is approached using a two phase technique. The first phase is a new robust fitting algorithm that overcomes some of the limitations of the Least Median of Squares robust technique. The second phase is an efficient relaxation style algorithm to refine the linear segments provided by the first phase to make second order estimates of the surface, and cluster estimates whose positions, orientations and curvatures make them consistent. Preliminary experimental results on one-dimensional synthetic data demonstrate the promise of the approach.

This paper presents a robust segmentation and fitting technique. The method randomly samples appropriate range image points and fits them into selected primitive type. From K samples we measure residual consensus to choose one set of sample points which determines an equation to have the best fit for a homogeneous patch in the current processing region. A method with compressed histogram is used to measure and compare residuals on various noise levels. The method segments range image into quadratic surfaces, and works very well even in smoothly connected regions.

This paper describes a general purpose, representation independent method for the accurate and computationally efficient registration of 3-D shapes including free-form curves and surfaces. The method handles the full six-degrees of freedom and is based on the iterative closest point (ICP) algorithm, which requires only a procedure to find the closest point on a geometric entity to a given point. The ICP algorithm always converges monotonically to the nearest local minimum of a mean-square distance metric, and experience shows that the rate of convergence is rapid during the first few iterations. Therefore, given an adequate set of initial rotations and translations for a particular class of objects with a certain level of 'shape complexity', one can globally minimize the mean-square distance metric over all six degrees of freedom by testing each initial registration. For examples, a given 'model' shape and a sensed 'data' shape that represents a major portion of the model shape can be registered in minutes by testing one initial translation and a relatively small set of rotations to allow for the given level of model complexity. One important application of this method is to register sensed data from unfixtured rigid objects with an ideal geometric model prior to shape inspection. The described method is also useful for deciding fundamental issues such as the congruence (shape equivalence) of different geometric representations as well as for estimating the motion between point sets where the correspondences are not known. Experimental results show the capabilities of the registration algorithm on point sets, curves, and surfaces.

A planetary rover will have various vision based requirements for navigation, terrain characterization, and geological sample analysis. In this paper we describe a knowledge-based controller and sensor development system for terrain analysis. The sensor system consists of a laser ranger and a CCD camera. The controller, under the input of high-level commands, performs such functions as multisensor data gathering, data quality monitoring, and automatic extraction of sample images meeting various criteria. In addition to large scale terrain analysis, the system's ability to extract useful geological information from rock samples is illustrated. Image and data compression strategies are also discussed in light of the requirements of earth bound investigators.

This paper presents a state-based control scheme for sensor fusion in autonomous mobile robots. States specify the sensing strategy for each sensor; the feedback rule to be applied to the sensors; and a set of failure conditions, which signal abnormal or inconsistent evidence. Experiments were conducted in the surveillance domain, where the robot was to determine if three different areas in a cluttered tool room remained unchanged after each visit. The data collected from four sensors (a Sony Hi8 color camcorder, a Pulnix black and white camera, an Inframetrics true infrared camera, and Polaroid ultrasonic transducers) and fused using the sensor fusion effects architecture (SFX) support the claims that the state-based control scheme produces percepts which are consistent with the scene being viewed, can improve the global belief in a percept, can improve the sensing quality of the robot, and it robust under a variety of conditions.

Passive Infrared sensors used for intrusion detection, especially those used on mobile robots, are vulnerable to false alarms caused by clutter objects such as radiators, steam pipes, windows, etc., as well as deliberately caused false alarms caused by decoy objects. To overcome these sources of false alarms, we are now combining thermal and ultrasonic signals, the results being a more robust system for detecting personnel. Our paper will discuss the fusion strategies used for combining sensor information. Our first strategy uses a statistical classifier using features such as the sonar cross-section, the received thermal energy, and ultrasonic range. Our second strategy uses s 3-layered neural classifier trained by backpropagation. The probability of correct classification and the false alarm rate for both strategies will be presented in the paper.

Traffic and transportation engineers continually require a more accurate and large amount of pedestrian flow data for numerous purposes. For example, the increasing use of pedestrian facilities such as building complexes, shopping malls, and airports in densely populated cities demands pedestrian flow data for planning, design, operation, and monitoring of these facilities. Currently, measurement of pedestrian flow data is often performed manually. This paper proposes a robot vision system to measure the number and walking direction of pedestrians using difference image and shape reconstruction techniques. The system consists of eight steps: (1) conversion of video images, (2) digitization of frozen frames, (3) conversion of 256-grey-level images into bilevel images, (4) extraction of rough sketch of pedestrian using difference images, (5) removal of line-noise, (6) reconstruction of shape of the pedestrian, (7) measurement of the number of pedestrians, and (8) determination of the direction of pedestrian movement. In this system, the operations in each step depend only on local information. Thus, they can be performed independently in parallel. A very large scale integration architecture can be implemented in this system to speed up calibration. The accuracy in measuring the number of pedestrians and their direction of travel is about 93% and 92%, respectively.

Space based materials processing experiments can be enhanced through the use of IVA robotic systems. A program to determine requirements for the implementation of robotic systems in a microgravity environment and to develop some preliminary concepts for acceleration control of small, lightweight arms has been initiated with the development of physical and digital simulation capabilities. The physical simulation facilities incorporate a robotic workcell containing a Zymark Zymate II robot instrumented for acceleration measurements, which is able to perform materials transfer functions while flying on NASA's KC-135 aircraft during parabolic maneuvers to simulate reduced gravity. Measurements of accelerations occurring during the reduced gravity periods will be used to characterize impacts of robotic accelerations in a microgravity environment in space. Digital simulations are being performed with TREETOPS, a NASA developed software package which is used for the dynamic analysis of systems with a tree topology. Extensive use of both simulation tools will enable the design of robotic systems with enhanced acceleration control for use in the space manufacturing environment.

This paper presents two examples of low level strategies using multisensor data fusion, one for bridge extraction, and one for urban area extraction. These extractions are made front a couple of coregistred Synthetic Aperture Radar (SAR) and SPOT images. These features are very different by their dimensions, their shape, and their radiometry. So we C1U prove the reliability of our approach on many types of features. Our method uses the notion of complementarity of each sensor, and the notion of context in the observed scene. For bridge detection, we first segment water in the SPOT image, to spatially constrain the bridge research in the SAR image. This research is achieved using a correlation method. To detect an urban area, we first use the knowledge that it produces very bright texture in SAR imagery. Thus, the main part of urban backscatters is extracted using an adaptative thresholding which keeps the upper band of the gray level histogram of the SAR image. This mask is then used for classification as a training set using a distance map of urban area texture in SPOT image. We determine the non urban zone training set using a distance map of the urban training zone boundaries. Classification is performed with multivariate Gaussian classifier. The results we obtained are very encouracting, especially if we consider the robustness of the bridge detection method.

This paper describes a recipe for the construction of control systems that support complex machines such as multi-limbed/multi-fingered robots. The robot has to execute a task under varying environmental conditions and it has to react reasonably when previously unknown conditions are encountered. Its behavior should be learned and/or trained as opposed to being programmed. The paper describes one possible method for organizing the data that the robot has learned by various means. This framework can accept useful operator input even if it does not fully specify what to do, and can combine knowledge from autonomous, operator assisted and programmed experiences.

Control of multiple robots has attracted researchers' interests for many years. Problems arise when uncertainty or coordinated motions are involved. This paper investigates several issues related to control of robot systems with multiple arms. We present a systematic control strategy for the coordination of robot motions during homogeneous manipulations. A hybrid force and position control law is used to determine the force and position targets of the object being manipulated. The frictional equilibrium and stability principles are employed in computing the optimal force distribution among the robots. The position targets for the end effector of individual robots are calculated using appropriate transformations which take into account the coordinating constraints. A generalized control algorithm is developed using this strategy and the manipulation can be achieved by specifying a number of external quantities, which describe the nature of the tasks to be performed, as the inputs to the control algorithm. This control strategy is adopted in developing control algorithms for heterogeneous manipulation tasks such as tasks that require robot interchange during the manipulation. In such as task, it is necessary to coordinate the motions between the coordinating robots and a free robot. A hierarchical control mechanism is used in achieving this coordination. The control primitives developed for controlling robots in the homogeneous manipulation can be used in this hierarchical control. The control algorithms have been successfully implemented on a robot system to perform various tasks with good results.

We extend the MRF image model commonly employed in the Bayesian development of image segmentation procedures to include a degradation channel resulting in a 2D hidden Markov model as the basis for the segmentation problem. We solve the segmentation problem by deriving the expectation-maximization algorithm for the case where the 'hidden' Markov source is the 2-D MRF that generates a true scene and the degradation channel is an additive, memoryless, grey-level degradation process that produces the observed scene.