Computer Vision for Student Learning

NSF-Sponsored Project (1551572)

"INT: Collaborative Research: Detecting, Predicting and Remediating Student Affect and Grit Using Computer Vision"

The objectives of the proposed project are designing

a real-time computer vision system for detecting the affect of students working with a computer tutor,
a machine learning system for predicting student grit (self-efficacy and persistence),
methods for augmenting an existing computer tutor so that it adapts to student needs and helps remediate low levels of student engagement and grit.

This is a collaborative project involving Boston University (PI Margrit Betke), and three other institutions, University of Massachusetts Amherst (Lead PI Beverly Woolf), Worcester Polytechnic Institute (PI Ivon Arroyo), and Clark University (PI John Magee). The project started in September 2016 and is projected to run until August 2022.

The research team at Boston University currently includes

Ajjen Joshi and Kevin Delgado are alumni.

BU CS Department Announcement, September 6, 2016.

Student Engagement Dataset

Student Data

Publications

Nataniel Ruiz, Mona Jalal, Hao Yu, Danielle Allessio, Ajjen Joshi, Thomas Murray, John Magee, Jacob Whitehill, Vitaly Ablavsky, Ivon Arroyo, Beverly Woolf, Stan Sclaroff, and Margrit Betke. Leveraging Affect Transfer Learning for Behavior Prediction in an Intelligent Tutoring System. IEEE International Conference on Automatic Face and Gesture Recognition 2021. December 2021. 8 pages.

K. Delgado, J. M. Origgi, T. Hasanpoor, H. Yu, D. Allessio, I. Arroyo, W.Lee, M. Betke, B. Woolf, and S. Adel Bargal. Student Engagement Dataset. ICCV 2021: 2nd Workshop and Competition on Affective Behavior Analysis in-the-wild (ABAW). October 2021. 9 pages.

H. Yu, A. Gupta, W. Lee, I. Arroyo, M. Betke, D. Allesio, T. Murray, J. Magee, B. P. Woolf Measuring and Integrating Facial Expressions and Head Pose as Indicators of Engagement and Affect in Tutoring Systems. In Adaptive Instructional Systems. Adaptation Strategies and Methods, Third International Conference, AIS 2021, Held as Part of the 23rd HCI International Conference, HCII 2021, Virtual Event, July 24–29, 2021, Proceedings, Part II, pp.219-233.

A. Joshi, D. Allessio, R. Lisle, J. Magee, I. Arroyo, S. Sclaroff, B. Woolf, and M. Betke. Affect-driven Learning Outcomes Prediction in Intelligent Tutoring Systems. The 14th IEEE Conference on Automatic Face and Gesture Recognition, Lille, France, May 2019, 5 pages. pdf.