Ellen,
We’ve done some work analyzing Zoom video recordings to look at facial expression, eye tracking, and some other cues. We’re using OpenFace and other open source tools.
When you do eye tracking, you have to calibrate at the start (“Hello, please look at me for a count of 5…now please look at this object on my PPT slide for a count of 5… thank you” etc.). Then you have constrain and manage Zoom some because if you don’t, the video tiles will change size and move around the screen as people share screen, stop sharing, etc. Or, they’ll be too small to analyze effectively. Using the Zoom Pro cloud recording feature is best because it will yield separate video recordings for the active speaker, gallery view, and shared screen.
Re measuring the higher-level constructs of checking out and multitasking, etc., you have to be careful about the mapping from your observable data to these. Is someone checked out or taking notes in another window? How many monitors do they have, and how big are they? Could they still be looking at the Zoom but just reading the chat briefly, or are they typing an email and not attending at all? This is a little harder to infer without full screen capture.
In one project we used OBS Studio (https://obsproject.com/) to record employees’ multiple monitors (with their permission) and then manually analyzed the recordings post hoc for behavioral codes using BORIS (https://www.boris.unito.it/). But we’re working on some more automatic analysis. Not sure how possible it will be.
Stephen
--
Stephen B. Gilbert, Ph.D. (he/him/his)
Director, Human Computer Interaction
Associate Director, Virtual Reality Applications Center
Associate Professor, Industrial and Manufacturing Systems Engineering
Iowa State University
515-294-6782
Original Message:
Sent: 3/25/2021 8:57:00 AM
From: Ellen Bass
Subject: Assessing engagement on video calls
EXTERNAL MESSAGE
________________________________
A colleague asked me the following: I'm curious whether you know of anyone actively thinking about using the video feed to actively track participants breathing rates, eye tracking, facial expressions, attention cues, engagement, etc. as a means to 1) identify/assess engagement levels, 2) actively enhance/improve engagement (prevent "checking out", multitasking, etc), and/or 3) as a means for measuring effects of testing interventions for such, e.g. in A/B testing.
Does anyone have pointers?
Best,
Ellen Bass