Gesture-Based Object Localization for Robot Applications in Intelligent Environments

Drawing attention to objects and their localization in the environment are essential building blocks for domestic robot applications, e.g. fetch-and-delivery or navigation tasks. For this purpose, human pointing gestures turned out to be a natural and intuitive interaction method to transfer the spatial data of an object from human to robot. Current approaches only use the robot's on-board sensors to perceive gesture-based instructions, which restricts them to the field of view of the robot's camera. The integration of mobile robots into intelligent environments, such as smart homes, opens new possibilities to overcome this limitation by utilizing components of the surrounding environment as additional sensors. We take advantage of these new possibilities and propose a multi-stage object localization system based on human pointing gestures that considers the whole intelligent environment as interaction partner. Our experimental results show that our multi-stage approach successfully refines the position initially proposed by a human pointing gesture by employing a distributed camera network integrated into the environment for object localization.