Summary

We have demonstrated a system which can respond to a user's face in real-time using completely passive and non-invasive techniques. Robust performance is achieved through the integration of three key modules: depth estimation to eliminate background effects, skin color classification for fast tracking, and face detection to discriminate the face from other body parts. Our system would be valuable for applications in interactive entertainment, telepresence/virtual environments, and intelligent kiosks which respond selectively according to the presence and pose of a user. We hope these and related techniques can eventually balance the I/O bandwidth between typical users and computer systems, so that they can control complicated virtual graphics objects and agents directly with their own expression.