MIT Neuroscientists Develop Model to Predict Human Emotions

The ability to anticipate and predict the emotions of others is a cognitive skill known as theory of mind. MIT neuroscientists have developed a computational model that can predict other people’s emotions, including joy, gratitude, confusion, regret, and embarrassment. The model was created based on the prisoner’s dilemma, a classic game theory scenario in which two people must decide whether to cooperate with their partner or betray them. The researchers incorporated several factors that have been hypothesized to influence people’s emotional reactions, including the person’s desires, expectations in a specific situation, and whether anyone is watching their actions.

The Study

The lead author of the study is Sean Dae Houlihan Ph.D. ’22, a postdoc at the Neukom Institute for Computational Science at Dartmouth College. The paper appeared today in Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences. Other authors include Max Kleiman-Weiner Ph.D. ’18, a postdoc at MIT and Harvard University; Luke Hewitt Ph.D. ’22, a visiting scholar at Stanford University; and Joshua Tenenbaum, a professor of computational cognitive science at MIT and a member of the Center for Brains, Minds, and Machines and MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL).

While previous research has focused on training computer models to infer someone’s emotional state based on their facial expression, the MIT model focuses on predicting someone’s emotional response to events before they occur. According to Rebecca Saxe, the John W. Jarve Professor of Brain and Cognitive Sciences and a member of MIT’s McGovern Institute for Brain Research, predicting emotions is the most critical aspect of human emotional intelligence.

To create a computational model that can predict emotions, the researchers used scenarios taken from a British game show called “Golden Balls.” The game involves contestants paired up with a pot of $100,000 at stake. After negotiating with their partner, each contestant decides, secretly, whether to split the pool or try to steal it. If both decide to split, they each receive $50,000. If one splits and one steals, the stealer gets the entire pot. If both try to steal, no one gets anything.

Depending on the outcome, contestants may experience a range of emotions. To create a computational model that can predict these emotions, the researchers designed three modules. The first module is trained to infer a person’s preferences and beliefs based on their action, through a process called inverse planning. Using this approach, the first module can predict contestants’ motivations based on their actions in the game.

The second module compares the outcome of the game with what each player wanted and expected to happen. Then, a third module predicts what emotions the contestants may be feeling, based on the outcome and what was known about their expectations. This third module was trained to predict emotions based on predictions from human observers about how contestants would feel after a particular outcome.

The model’s success stems from its incorporation of key factors that the human brain also uses when predicting how someone else will react to a given situation, Saxe says. Those include computations of how a person will evaluate and emotionally react to a situation, based on their desires and expectations, which relate to not only material gain but also how they are viewed by others.

The researchers hope to adapt the model so that it can perform more general predictions based on situations other than the game-show scenario used in this study. They are also working on creating models that can predict what happened in the game based solely on the expression on the faces of the contestants after the results were announced. The model is a significant step forward in developing a deeper understanding of how emotions contribute to determining our actions, and it explains how we can use people’s actions to infer their underlying emotions.