Facial Feature Tracking and Occlusion Processing in American Sign Language
Thomas J. Castelli
|Margrit Betke||Carol Neidle|
|Department of Electrical and Computer Engineering||Computer Science Department||Department of Modern Foreign Languages|
|8 Saint Mary's Street||111 Cummington Street||718 Commonwealth Avenue|
|Boston, MA 02215|
Facial features play an important role in expressing grammatical information
in signed languages, including American Sign Language (ASL). Gestures such
as raising or furrowing the eyebrows are key indicators of constructions such as
yes-no questions. Periodic head movements (nods and shakes) are also an
essential part of the expression of syntactic information, such as negation
(associated with a side-to-side headshake). Therefore, identification of these
facial gestures is essential to sign language recognition. One problem
with detection of such grammatical indicators is occlusion recovery. If
the signer's hand blocks his/her eyebrows during production of a sign, it
becomes difficult to track the eyebrows. We have developed a system to
detect such grammatical markers in ASL that recovers promptly from occlusion.
Our system detects and tracks evolving templates of facial features, which are based on an anthropometric face model, and interprets the geometric relationships of these templates to identify grammatical markers. It was tested on a variety of ASL sentences signed by various Deaf native signers, and detected facial gestures used to express grammatical information, such as raised and furrowed eyebrows as well as headshakes.
Below are links to five AVI video clips showing our system processing video of ASL signers. The status bar (just under the video) shows which grammatical markers have been detected by the system for that frame.
Video Clips (videos will come soon)
Sign order: FATHER FUTURE LIKE THAT BOOK
Sign order: FATHER GIVE JOHN HOW-MANY BOOK
Sign order: JOHN SEE THROW APPLE WHO ... MARY
Sign order: JOHN FUTURE NOT BUY HOUSE
Sign order: MARY SELF PREFER CORN
We would like to thank Stan Sclaroff and Vassilis Athitsos for helping to provide the ASL videos used in this research.
Funding was provided by the National Science Foundation (IIS-0329009, IIS-0093367, IIS-9912573, EIA-0202067, and EIA-9809340) and the Office of Naval Research (N000140110444).