CS 591 Seminar

Video-Based Human-Computer Interfaces

Fall 2001

Seeing Me in My Office:
Please feel free to stop by my office anytime. My office is in MCS 286 (111 Cummington St). I am generally around every day. I may be in meetings, so the best time to reach me is during office hours. You can also make an appointment by email or by speaking with me before class (I don't have time after class, since I have to rush to another class). I'm happy to talk with you about the course, computer vision research, your plans for the future, or anything else. I hope that we will develop a good relationship as the term progresses. Check out my personal web page to get to know me a little.

Our goal is to learn about computer systems that analyze images automatically, determine what the computer ``sees'' or ``recognizes,'' and thus provide new interfaces for human users.
We will read and discuss papers on video-based human-computer interfaces and maybe have some outside speakers and go for a field trip to a research lab in the area. The updated course syllabus and some handouts are made available on our course web page.
There are no formal prerequisites, but a computer vision or graphics background would be great. If you have not taken cs585 or cs480, you may want to pick up a textbook and do some extra reading (e.g., Robot Vision by BKP Horn, or Machine Vision by Jain, Kasturi, and Schunck, both books published by The MIT Press).

A textbook is not required. Reading materials will be handed out before lectures. If you need computer access, use the Computer Science Department's Undergraduate Lab located at 730 Commonwealth Ave, 3rd floor. Please add yourself to the course mailing list by typing csmail -a cs591a1 on your cs account. You can also use the Human-Computer Interface Laboratory at 64 Cummington St.

Class Participation: Come to class and participate regularly. Reading the assigned papers and listening in class will only give you a "passive understanding" of the material. I encourage discussions in class to help you acquire an "active understanding" of the material so that you can evaluate existing human-computer interfaces critically and learn to develop your own creative solutions.

Reading: To prepare for each class, you will be asked to read book chapters, conference articles, journal papers, and on-line publications. You can only achieve a good understanding and appreciation of the state-of-the-art of video-based human-computer interfaces if you read the assigned papers consistently and thoroughly.

Class Presentations: You will be asked to present a paper, a set of papers, or a project to the class and lead the discussion in class. You can follow the outline of the paper or choose your own. Introduce the topic of the work, explain the long-term and short-term goals, and discuss the authors' motivation. Explain the methods, algorithms, and techniques. How do they relate to previous work? You may need to discuss some additional background material. Report on the experiments and results. Are they satisfactory? What are the authors' conclusions? What are your own conclusions? Give a critical evaluation. Is this promising work? Has it been successful? What are the limitations?
See me in my office hours to discuss the paper well in advance of your presentation.

Voluntary Project: Students who enjoy programming and are interested in building video-based human-computer interfaces are encouraged to design or participate in a project. Projects are voluntary. You can substitute a research paper presentation in class with a write-up and presentation of your project. You can also use it to fullfill the project requirement in the BA/MA and MA programs. You can use the new Human-Computer Interface Laboratory in 64 Cummington Street to develop your project.

On-line Class Notes: You will be asked to prepare on-line class notes for one or two class presentations by other students. Notes taken during a Monday class are due the following Thursday in my 2:30-3:30 pm office hours. Notes taken during a Wednesday class are due the following Monday in my 2:00-3:00 pm office hours. During these office hours, I will go through the notes with you and maybe suggest changes. If you cannot come to the office hours, email me the source code before the deadline. After you have made the corrections (if needed), I will publish the notes on our course web page. Your timelyness is important. Late class notes will be levied a penalty of 20% per day (up to three days).

Exams: There will be two in-class exams on the reading materials assigned and discussed in the class. The exams will be quite easy for students who kept up with assigned readings, paper presentations, and discussions. The dates of the exams are Monday, October 15, and Wednesday, December 5. You are allowed to use one double-sided page of notes in each exam. There will be no make-up exams. There is no final exam.

Grading Policy: Your final grade will be determined roughly as follows:

• Class participation 15%
• Class presentations 25%
• On-line class notes 10%
• Each exam 25%

You are encouraged to collaborate with classmates on the preparation class presentations, lecture notes, and projects. If you do, you must acknowledge your collaborators. Cheating during exams is not worthy of Boston University students. I expect you to abide by the standards of academic honesty and computer ethics policy described in http://www.bu.edu/computing/ethics/ .

CS591 is an elective seminar course that will introduce you to an exciting topic in computer science. It should be fun and not too much of a struggle for you. If you do not understand the material, ask questions in class. If one student is confused about something, then maybe others are also confused and grateful that someone asked. Also, come and see me for help or send me email.

Course Schedule

The syllabus may change depending on the time spent on different topics and whether or not new or alternative papers are discussed. Topic and paper suggestions are welcome. We will schedule a field trip after a discussion in class about possible destinations.

Reading List

M 9/10 -- Instructor

• Smart Rooms: Machine Understanding of Human Behavior, A. Pentland. In R. Cipolla and A. Pentland, editors, "Computer Vision for Human-Machine Interaction," 1998. Cambridge University Press. Handout.
• MIT Media Lab's web page on "smart rooms." (Optional)
• Scribe: Bill Mullally. Lecture notes: protected pdf protected ps

• The Camera Mouse: Visual Tracking of Body Features to Provide Computer Access For People with Severe Disabilities, M. Betke J. Gips, and P. Fleming, August 2000, pdf, ps. protected pdf. protected ps.
• The Camera Mouse: Preliminary Investigation of Automated Visual Tracking for Computer Access, J. Gips, M. Betke, P. Fleming. In the Proceedings of the Rehabilitation Engineering and Assistive Technology Society of North America 2000 Annual Conference, RESNA 2000, pp. 98-100, Orlando, FL, July 2000, pdf. protected pdf. (Optional)
• Early Experiences Using Visual Tracking for Computer Access by People with Profound Physical Disabilities, J. Gips, M. Betke, and P. A. DiMattia. In the Proceedings of the 1st International Conference on Universal Access in Human-Computer Interaction, UAHCI 2001 , New Orleans, LA, August 2001, pdf, protected pdf (Optional)
• Scribe: Johanna Brewer. Lecture notes: protected pdf,

• Active Detection of Eye Scleras in Real Time, M. Betke, W. Mullally, J. Magee. Proceedings of the IEEE CVPR Workshop on Human Modeling, Analysis and Synthesis, HMAS 2000, Hilton Head Island, SC, June 2000. ps, ps.gz.
• Elliptical Head Tracking Using Intensity Gradients, S. Birchfield, Proceedings of the IEEE Computer Vision and Pattern Recognition Conference, pp. 232-237, Hilton Head Island, SC, 2000. protected pdf
• Tracking Faces, A. H. Gee and R. Cipolla. In R. Cipolla and A. Pentland, editors, "Computer Vision for Human-Machine Interaction," 1998. Cambridge University Press. Handout.
• An Algorithm for Real-time Stereo Vision Implementation of Head Pose and Gaze Direction Measurement. Y. Matsumoto and A. Zelinsky. In Proceedings of the International Conference on Face and Gesture Recognition, pp. 499-504, 2000. protected pdf.
• Interview on Australian TV about an Eye Tracker, April 2001. (Optional)
• Scribes: Bill Mullally, protected ps, protected pdf, and Yannis Minadakis: protected pdf.

• Fast, Reliable Head Tracking under Varying Illumination: An Approach Based on Robust Registration of Texture-Mapped 3D Models, M. La Cascia, S. Sclaroff, and V. Athitsos, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(4), April 2000. pdf.
• Scribe: Chekema Prince Lecture notes: protected pdf.

• Measuring Facial Movement with the Facial Action Coding System, Paul Ekman and Wallace V. Friesen. In "Emotion in the human face," edited by Paul Ekman, 1982. Cambridge University Press. Handout.
• Review of Research, 1970-1980, Paul Ekman and Harriet Oster. In "Emotion in the human face," edited by Paul Ekman, 1982. Cambridge University Press. Handout.
• Tracking Facial Motion, I. Essa, T. Darrell, and A. Pentland. In the Proceedings of the IEEE Nonrigid and Articulated Motion Workshop, 1995. pdf. ps.
• Recognizing Facial Expressions in Image Squences Using Local Parameterized Models of Image Motion, M. Black and Y. Yacoob. Int. Journal of Computer Vision, 25(1), pp. 23-48, 1997. Also Xerox PARC Technical Report SPL-95-020, March 1995. protected ps (Optional)
• Scribes: Seema Shah: protected pdf, and Stephen Crampton: protected pdf.

• Interfacing through Visual Pointers, C. Colombo, A. Del Bimbo, and S. De Magistris. In R. Cipolla and A. Pentland, editors, "Computer Vision for Human-Machine Interaction," 1998. Cambridge University Press. Handout.
• What you look at is what you get, R. J. K. Jacob. Computer, pp. 65-66, July 1993. protected pdf
• Scribe: Robyn Cloud protected pdf.

• Pupil detection and tracking using multiple light sources, C. Morimoto, D. Koons, A. Amit, M. Flickner and S. Zhai. IBM Almaden Research Center, Report RJ-10177, 1998. protected ps protected pdf
• Keeping an eye for HCI, C. Morimoto, D. Koons, A. Amit, M. Flickner and S. Zhai. Proceedings of the XII Brazilian Symposium on Computer Graphics and Image Processing, pp. 171-176, 1999. protected pdf
• IBM's BlueEyes project web page.
• K. Grauman, M. Betke, J. Gips, G. R. Bradski, "Communication via Eye Blinks - Detection and Duration Analysis in Real Time," Proceedings of the IEEE Computer Vision and Pattern Recognition Conference CVPR 2001, Kauai, Hawaii, December 2001. protected pdf, protected ps.
• Software used in blink experiments: See free demo of "Scan & Match Series" program by Simtech Publications.
• Intel's eyeblink patent. Read "Description" part. (Optional)
• Scribes: Bevan Lew: protected pdf, Walter Chang: protected pdf

• Neural Networks, Linear Algebra, Multivariate Gaussians
• Scribe: Jason Peterson, Lecture notes on line soon.

• Example-based learning for view-based human face detection, T. Poggio and K.-K. Sung. Proceedings of the ARPA IU Workshop '94, II, pp. 843 - 850, 1994. pdf ps.
• CMU Interactive Demonstration of a Face Detector, web page. (Optional)
• Scribe: Evelina Prokupets. pdf.

• Low-dimensional procedure for the characterization of human faces, L. Sirovich and M. Kirby. Journal of the Optical Society of America A, 4:3, March 1987. Handout.
• Eigenfaces for recognition, M. Turk, and A. Pentland. Journal of Cognitive Neuroscience, 3(1), 71-86. 1991. Handout. (Optional)
• Application of the Karhunen-Loeve Procedure for the Characterization of Human Faces, M. Kirby and L. Sirovich. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12:1, January 1990. pdf (Optional)
• MIT Media Lab Face Recognition Projects, web page. (Optional)

• 3D Trajectory Recovery for Tracking Multiple Objects and Trajectory Guided Recognition of Actions, R. Rosales and S. Sclaroff. In Proceedings IEEE Computer Vision and Pattern Recognition (CVPR), Fort Collins, CO, 1999. pdf, ps.

• Introduction to Hidden Markov Models

• The Kidsroom: A Perceptually-Based Interactive and Immersive Story Environment, A. Bobick, S. Intille, J. Davis, F. Baird, C. Pinhanez, L. Campbell, Y. Ivanov, A. Schutte, and A. Wilson. In Presence: Teleoperators and Virtual Environments, 8(4), 1999, pp. 367-391. pdf, ps.Z.
• The Kids Room. An Interactive Narrative Playspace. MIT Media Lab.
• Scribe: Witold Stankiewicz html.

• The Recognition of Human Movement Using Temporal Templates, A. Bobick and J. Davis, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 23, No. 3, 2001, pp. 257-267. pdf.
• Model-based estimation of 3D human motion, I. Kakadiaris and D. Metaxas, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22:12, pp. 1453 -1459, December 2000. pdf.
• W4 : Who? When? Where? What ? A Real Time System for Detecting and Tracking of People, I. Haritaoglu, L. Davis. University of Maryland, web page. (Optional)
• Scribes: John Magee. pdf. Jonathan Lombardi. pdf.

• A Framework for Gesture Generation and Interpretation, J. Cassell. In R. Cipolla and A. Pentland, editors, "Computer Vision for Human-Machine Interaction," 1998. Cambridge University Press.
• Scribe: Anna Karpovsky. html.

• GestureComputer - History, Design and Applications, C. Maggioni and B. Kammerer. In R. Cipolla and A. Pentland, editors, "Computer Vision for Human-Machine Interaction," 1998. Cambridge University Press. Handout.
• Visual Sensing of Humans for Active Public Interfaces K. Waters, J. Rehg, M. Loughlin, S. B. Kang, and D. Terzopoulos, in R. Cipolla and A. Pentland, editors, "Computer Vision for Human-Machine Interaction," 1998. Cambridge University Press. Handout.
• Model-Based Interpretation of Faces and Hand Gestures, C. J. Taylor, A. Lanitis, T. F. Cootes, G. Edwards, and T. Ahmad. In R. Cipolla and A. Pentland, editors, "Computer Vision for Human-Machine Interaction," 1998. Cambridge University Press. Handout.
• Scribe for first two papers: Raymond Peng. 1st paper: pdf. 2nd paper: pdf.
• Scribe for last paper: Jingbin Wang pdf.

• Pupil Detection and Tracking (no paper)
• Preparation for Field Trip
• Scribe: Jason Ruel. html.

• Field Trip: Museum of Science, Computer Exhibit
• Test the software of your choice. Would a video-based human-computer interface, for example, the CameraMouse or the BlinkLink, be useful for your software? How could it be incorporated?

• Finger Counting, S. Crampton, unpublished manuscript, 2001. pdf.
• No scribe. See paper.

• Introduction to American Sign Language (ASL) web page.
• ASL Recognition Based on a Coupling Between HMMs and 3D Motion Analysis, C. Vogler and D. Metaxas, "ICCV," pp. 363-369, Mumbai, India, 1998. ps, pdf.
• Toward Scalability in ASL Recognition: Breaking Down Signs into Phonemes, C. Vogler and D. Metaxas, "Gesture Workshop," France, 1999. ps, pdf. (Optional)
• Scribes: George Valavanis - pdf, Dan Hoeweler - html.

• A New 3D Lip Model for Analysis and Synthesis of Lip Motion in Speech Production, L. Reveret and C. Benoit. In the Proceedings of the Second ESCA Workshop on Audio-Visual Speech Processing, AVSP'98, Terrigal, Australia, Dec. 4-6, 1998. pdf, ps.
• Probabilistic Models of Verbal and Body Gestures C. Bregler, S. M. Omohundro, M. Covell, M. Slaney, S. Ahmad, D. A. Forsyth and J. A. Feldman. In R. Cipolla and A. Pentland, editors, "Computer Vision for Human-Machine Interaction," 1998. Cambridge University Press. Handout.
• Video Rewrite: Driving Visual Speech with Audio, web page. (Optional)
• Sound Localization Research at SJSU, web page. (Optional)
• Scribes: Harrison Hong pdf and ?

• Real-Time Visual Tracking of the Surgeon's Face for Laparoscopic Surgery, A. Nishikawa, T. Hosoi, K. Koara, D. Negoro, A. Hikita, S. Asano, F. Miyazaki, M. Sekimoto, Y. Miyake, M. Yasui, and M. Monden. In Medical Image Computing and Computer-Assisted Intervention - MICCAI 2001, Proceedings of the 4th International Conference, Utrecht, The Netherlands, W. J. Niessen and M. A. Viergever, editors, Springer, October 2001.
• Image Guided Surgery, web page. (Optional)
• Scribe: Jingbin Wang. pdf

• Wearable Cinema/Wearable City: bridging physical and virtual spaces through wearable computing, F. Sparacino, G. Davenport and A. Pentland, IMAGINA 2000, Montecarlo, January 2000. pdf.
• The Beware Home: A Contextually Aware Haunted House, C. D. Kidd, T. Starner, M. Gandy, and A. Quay. Technical Report GIT-GVU-00-29, Georgia Institute of Technology, 2000. pdf, ps.Z.
• The challenges of wearable computing: Part 1, T. Starner, IEEE Micro, 21:4, pp. 44 - 52, July-Aug. 2001 pdf. (Optional)
• The challenges of wearable computing: Part 2, T. Starner, IEEE Micro, 21:4, pp. 54 - 67, July-Aug. 2001 pdf. (Optional)
• Humanistic computing: "WearComp" as a new framework and application for intelligent signal processing, S. Mann, Proceedings of the IEEE 86(11), pp. 2123 -2151, November 1998. pdf. (Optional)
• Scribe: Joo-Hee Kim: 1st paper: pdf, 2nd paper: pdf.

Computer Vision Links

Check out http://www.cs.bu.edu/faculty/betke/links.html. You will find a list of links to computer vision conferences, journals, research groups, and companies.

Margrit Betke, Assistant Professor Computer Science Department Boston University 111 Cummington Street ( campus map ) Boston, MA 02215 Email: betke@cs.bu.edu URL: http://www.cs.bu.edu/faculty/betke Phone: 617-353-6412 Fax: 617-353-6457 Last updated: November 26, 2001