Interactive Inference of Mental States

Understanding a child’s social emotional state throughout an educational activity is crucial for a positive learning outcome. As such, we develop machine learning models that help social robots understand the cognitive states of children. Current state-of-the art approaches design inference models as passive processes of observing the nonverbal expressions of people. But these approaches are missing how those expressions are embedded in a greater interaction context. Our partially observable Markov decision process (POMDP) model captures how the meaning of an expression changes based on the cue context. This allows us to more accurately estimate a listener’s level of attention and engagement in a peer-to-peer storytelling interaction. We model social robots as having an active role in the inference process by generating social cues that elicit nonverbal responses from children to better understand their underlying cognitive state.

POMDP Model

Our partially observable Markov decision process (POMDP) model captures how the meaning of a listener’s response depends on the storyteller’s cue when estimating the cognitive state of the listener.

Publications

J.J. Lee and C. Breazeal (2014). Interactively Learning Nonverbal Behavior for Inference and Production: A Machine Learning Approach. Proceedings of the Algorithmic HRI Workshop at the ACM/IEEE International Conference on Human-Robot Interaction (HRI).

Interactive Inference of Mental States