This project involves research on computer-based recognition of ASL signs. One goal is development of a "look-up" capability for use as part of an interface with a multi-media sign language dictionary. Although printed dictionaries exist for ASL, they are generally organized according to the closest English translation of the ASL sign, since there is no written form for ASL. There are obvious problems resulting from the fact that there is no one-to-one correspondence between English words and ASL signs (imagine if you could only get information about French words--or words in any other spoken language--by looking them up under their English translations). This also poses an insurmountable difficulty for language learners, a kind of Catch-22: you can only look up a sign you don’t know in the dictionary if you already know what it means.

The proposed system will enable a signer either to select a video clip corresponding to an unknown sign, or to produce a sign in front of a camera, for look-up. The computer will then find the best match(es) from its inventory of thousands of ASL signs. Knowledge about linguistic constraints of sign production will be used to improve recognition. Fundamental theoretical challenges include the large scale of the learning task (thousands of different sign classes), the availability of very few training examples per class, and the need for efficient retrieval of gesture/motion patterns in a large database.

In addition to use with multi-media dictionaries, this technology will have many other applications, e.g., for computer-based automatic translation. This project includes plans to develop a “sloogle,” to do google-like searches through streams of ASL video.

This effort is part of the American Sign Language Linguistic Research Project at Boston University, in collaboration with Vassilis Athitsos at University of Texas at Arlington.

This web page describes research that is supported by the National Science Foundation, through grant 0705749. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

Publicly-Available Datasets

• American Sign Language Lexicon Video Dataset (ASLLVD)
• American Sign Language Linguistic Research Project Data