Video-Based Human-Computer Interfaces

The Camera Mouse

M. Betke J. Gips, and P. Fleming, "The Camera Mouse: Visual Tracking of Body Features to Provide Computer Access For People with Severe Disabilities." IEEE Transactions on Neural Systems and Rehabilitation Engineering, 10:1, pp. 1-10, March 2002. pdf.

J. Gips, P. DiMattia, M. Betke, "Collaborative Development of New Access Technology and Communication Software," 10th Biennial Conference of the International Society for Augmentative and Alternative Communication, ISAAC 2002, Odense, Denmark, August 2002, pdf.

J. Gips, M. Betke, and P. A. DiMattia, "Early Experiences Using Visual Tracking for Computer Access by People with Profound Physical Disabilities." 1st International Conference on Universal Access in Human-Computer Interaction, UAHCI 2001 , New Orleans, LA, August 2001.

J. Gips, M. Betke, P. Fleming, "The Camera Mouse: Preliminary Investigation of Automated Visual Tracking for Computer Access." Proceedings of the Rehabilitation Engineering and Assistive Technology Society of North America 2000 Annual Conference (RESNA) , Orlando, FL, July 2000.

C. Fagiani, M. Betke, and J. Gips, "Evaluation of Tracking Methods for Human-Computer Interaction." Proceedings of the IEEE Workshop on Applications in Computer Vision, WACV 2002, pp. 121-126, Orlando, Florida, December 2002.

Abstract. Tracking methods are evaluated in a real-time feature tracking system used for human-computer interaction (HCI). The Camera Mouse, a HCI system for people with severe disabilities that interprets video input to manipulate the mouse pointer was improved and used as the test platform for this study.
Tracking methods tested are the Lucas-Kanade tracker and a tracker based on normalized correlation. Both methods are evaluated with and without multidimensional Kalman filters. Two-, four-, and six-dimensional filters are tested to model feature location, velocity, and acceleration. The various tracker and filter combinations are evaluated for accuracy, computational efficiency, and practicality. The normalized correlation coefficient tracker without Kalman filtering is found to be the tracker best suited for a variety of human-computer interaction tasks.

R. L. Cloud, M. Betke, J. Gips, "Experiments with a Camera-Based Human-Computer Interface System." Proceedings of the 7th ERCIM Workshop "User Interfaces for All," UI4ALL 2002, pp. 103-110, Paris, France, October 2002.

Abstract. A human-computer interface (HCI) system called "The Camera Mouse" is evaluated. It tracks a user's movements with a video camera and translates them to movements of the mouse pointer on the screen. The main objectives for the experimentation were to quantitatively define the performance of the system for different users, features, and applications, to determine the optimal settings for different kinds of users, and to compare measurements over all users in order to suggest enhancements to a future system of this type. The experiments were conducted with 11 participants including a subject with severe physical disabilities. Each subject repeatedly performed a number of tasks. During each trial, a different feature was tracked and the elapsed time and mouse movement trajectories were measured. These measurements were used to quantify the system's performance.

Games that can be used with the Camera Mouse:

• Bouncer

Video Analysis for American Sign Language Recognition

T. Castelli, M. Betke, and C. Neidle, "Facial Feature Tracking and Occlusion Processing in American Sign Language"

Please visit our ASL web page.

The Blink Link

K. Grauman, M. Betke, J. Lombardi, J. Gips, and G. Bradski, "Communication via Eye Blinks and Eyebrow Raises: Video-Based Human-Computer Interfaces." Universal Access in the Information Society, 2(4), 359-373, November 2003.

K. Grauman, M. Betke, J. Gips, G. R. Bradski, "Communication via Eye Blinks - Detection and Duration Analysis in Real Time," Proceedings of the IEEE Computer Vision and Pattern Recognition Conference CVPR 2001, Kauai, Hawaii, December 2001.

Click here for video demos: Blink Detection Abstract. We developed a real-time vision system, the "Blink Link," that automatically detects a user's eye blinks and accurately measures their durations. The system is intended to provide an alternate input modality to allow people with severe disabilities to access a computer. Voluntary long blinks trigger mouse clicks, while involuntary short blinks are ignored. The system enables communication using "blink patterns:" sequences of long and short blinks which are interpreted as semiotic messages. The location of the eyes is determined automatically through the motion of the user's initial blinks. Subsequently, the eye is tracked by correlation across time, and appearance changes are automatically analyzed in order to classify the eye as either open or closed at each frame. No manual initialization, special lighting, or prior face detection is required. The system has been tested with interactive games and a spelling program. Results demonstrate overall detection accuracy of 95.6% and an average rate of 28 frames per second.

The Eyebrow Clicker

J. Lombardi and M. Betke, "A Camera-based Eyebrow Tracker for Hands-free Computer Control via a Binary Switch." Accepted at the 7th ERCIM Workshop "User Interfaces for All," UI4ALL 2002, Paris, France, October 2002.

Abstract. We designed the Eyebrow-Clicker, a camera-based human computer interface system that implements a new form of binary switch. When the user raises his or her eyebrows, the binary switch is activated and a selection command is issued. The Eyebrow-Clicker thus replaces the ``click'' functionality of a mouse. The system initializes itself by detecting the user's eyes and eyebrows, tracks these features at frame rate, and recovers in the event of errors. The initialization uses the natural blinking of the human eye to select suitable templates for tracking. Once execution has begun, a user therefore never has to restart the program or even touch the computer. In our experiments with human-computer interaction software, the system successfully determined 93% of the time when a user raised his eyebrows.

The Finger Counter

S. C. Crampton and M. Betke, "Finger Counter: A Human-Computer Interface." Accepted at the 7th ERCIM Workshop "User Interfaces for All," UI4ALL 2002, pp. 195-196, Paris, France, October 2002.

S. Crampton, M. Betke, "Counting Fingers in Real Time: A Webcam-Based Human-Computer Interface with Game Applications," Proceedings of the Conference on Universal Access in Human-Computer Interaction, affiliated with HCI International 2003, pp. 1357-1361, Crete, Greece, June 2003. pdf.

Abstract. Finger Counter is a computer-vision system that counts the numbers of fingers held up in front of a video camera in real time. The system is designed as a simple and universal human-computer interface: potential applications include educational tools for young children and supplemental input devices, particularly for persons with disabilities. The interface is language independent and requires minimal education and computer literacy. Finger Counter uses background differencing and edge detection to locate the outline of the hand. The system then processes the polar-coordinate representation of the pixels on the outline to identify and count fingers: fingers are recognized as protrusions that meet particular threshold requirements. The system also logs the frequency of different inputs over a given time interval. We implemented the Finger Counter interface under Linux using Video4Linux and also under Microsoft Windows as a DirectShow filter. The system was tested extensively under various lighting and background conditions. During testing, the system successfully counted the fingers of numerous subjects with disparate hand shapes and sizes and skin color. Finally, we incorporated the Finger Counter interface into a children's game for learning and entertainment.

Active Eye Detection and Tracking

M. Betke, B. Mullally, J. Magee, "Active Detection of Eye Scleras in Real Time." Proceedings of the IEEE CVPR Workshop on Human Modeling, Analysis and Synthesis, Hilton Head Island, SC, June 2000.

We developed an eye detection system that actively controls the camera's pan, tilt, and zoom that detects the white visible portion of the eyeball.

M. Betke and J. Kawai, "Gaze Detection via Self-Organizing Gray-Scale Units." Proceedings of the International Workshop on Recognition, Analysis, and Tracking of Faces and Gestures in Real-Time Systems, pp. 70-76, Kerkyra, Greece, September 1999.

EyeKeys

John J. Magee, Matthew R. Scott, Benjamin N. Waber, and Margrit Betke

Abstract. Some people are so severely paralyzed that they only have the ability to control the muscles in their eyes. For these people, eye movements or blinks are the only way to communicate. Currently available computer interface systems are often intrusive, require special hardware, or use active infrared illumination. An interface system called EyeKeys is presented. EyeKeys runs on a consumer grade computer with video input from an inexpensive USB camera. The face is tracked using multi-scale template correlation. Symmetry between left and right eyes is exploited to detect if the computer user is looking at the camera, or off to the left or right side. The detected eye direction can then be used to control applications such as spelling programs or games. The game ``BlockEscape'' was developed to gather quantitative results to evaluate our interface system with test subjects. The system is compared to a mouse substitution interface.

Our Collaborative Efforts

• www.cameramouse.org
• BC's EagleEyes
• Check out Rick Hoydt's page: www.teamhoyt.com
• The Boston Home