Computer Vision for Student Learning

NSF-Sponsored Project (1551572)

"INT: Collaborative Research: Detecting, Predicting and Remediating Student Affect and Grit Using Computer Vision"

The objectives of the proposed project are designing

a real-time computer vision system for detecting the affect of students working with a computer tutor,
a machine learning system for predicting student grit (self-efficacy and persistence),
methods for augmenting an existing computer tutor so that it adapts to student needs and helps remediate low levels of student engagement and grit.

This is a collaborative project involving Boston University (PI Margrit Betke), and three other institutions, University of Massachusetts Amherst (Lead PI Beverly Woolf), Worcester Polytechnic Institute (PI Ivon Arroyo), and Clark University (PI John Magee). The project started in September 2016 and is projected to run until August 2022.

The research team at Boston University currently includes

Ajjen Joshi and Kevin Delgado are alumni.

BU CS Department Announcement, September 6, 2016.

Student Engagement Dataset

Student Data

Publications

Hao Yu, Danielle A. Allessio, Will Lee, William Rebelsky, Frank Sylvia, Tom Murray, John J. Magee, Ivon Arroyo, Beverly P. Woolf, Sarah Adel Bargal, and Margrit Betke. COVES: A Cognitive-Affective Deep Model that Personalizes Math Problem Difficulty in Real Time and Improves Student Engagement with an Online Tutor. 9 pages. Accepted at the 31st ACM International Conference on Multimedia 2023. Ottawa, Canada, October 29-November 2, 2023.

Nataniel Ruiz*, Hao Yu*, Danielle A. Allessio, Mona Jalal, Ajjen Joshi, Tom Murray, John J. Magee, Kevin Manuel Delgado, Vitaly Ablavsky, Stan Sclaroff, Ivon Arroyo, Beverly P. Woolf, Sarah Adel Bargal, and Margrit Betke. ATL-BP: A Student Engagement Dataset and Model for Affect Transfer Learning for Behavior Prediction. IEEE Transactions on Biometrics, Behavior, and Identity Science. 5(3): 411-424. July 2023. DOI 10.1109/tbiom.2022.3210479. *Equal contribution.

Will Lee, Danielle Allessio, William Rebelsky, Sai Satish Gattupalli, Hao Yu, Ivon Arroyo, Margrit Betke, Sarah Bargal, Tom Murray, Frank Sylvia, and Beverly P. Woolf. Measurements and Interventions to Improve Student Engagement through Facial Expression Recognition. Edited by Robert A. Sottilare, Jessica Schwarz: Adaptive Instructional Systems - 4th International Conference, AIS 2022, Held as Part of the 24th HCI International Conference , HCII 2022, Virtual Event, June 26 - July 1, 2022, Proceedings. Lecture Notes in Computer Science 13332, Springer 2022, ISBN 978-3-031-05886-8 Pages 286--301.

Nataniel Ruiz, Mona Jalal, Hao Yu, Danielle Allessio, Ajjen Joshi, Thomas Murray, John Magee, Jacob Whitehill, Vitaly Ablavsky, Ivon Arroyo, Beverly Woolf, Stan Sclaroff, and Margrit Betke. Leveraging Affect Transfer Learning for Behavior Prediction in an Intelligent Tutoring System. IEEE International Conference on Automatic Face and Gesture Recognition 2021. December 2021. 8 pages.

K. Delgado, J. M. Origgi, T. Hasanpoor, H. Yu, D. Allessio, I. Arroyo, W.Lee, M. Betke, B. Woolf, and S. Adel Bargal. Student Engagement Dataset. ICCV 2021: 2nd Workshop and Competition on Affective Behavior Analysis in-the-wild (ABAW). October 2021. 9 pages.

H. Yu, A. Gupta, W. Lee, I. Arroyo, M. Betke, D. Allesio, T. Murray, J. Magee, B. P. Woolf Measuring and Integrating Facial Expressions and Head Pose as Indicators of Engagement and Affect in Tutoring Systems. In Adaptive Instructional Systems. Adaptation Strategies and Methods, Third International Conference, AIS 2021, Held as Part of the 23rd HCI International Conference, HCII 2021, Virtual Event, July 24–29, 2021, Proceedings, Part II, pp.219-233.

A. Joshi, D. Allessio, R. Lisle, J. Magee, I. Arroyo, S. Sclaroff, B. Woolf, and M. Betke. Affect-driven Learning Outcomes Prediction in Intelligent Tutoring Systems. The 14th IEEE Conference on Automatic Face and Gesture Recognition, Lille, France, May 2019, 5 pages. pdf.