Think of the last time you smiled. Was it because of a joke a friend told, or were you posing for a photo? Maybe it was to appear polite, or as part of greeting someone new. Each of these scenarios results in essentially the same expression, but with varying emotions behind each. So, if the same facial expression can mean different things, what can the face tell us about how we feel? It’s a complicated story.
To a machine, each of these expressions looks the same – a person’s lips stretch into a smile, their cheeks rise – it looks like an expression of joy. A machine could see these expressions and conclude: the person must be feeling happy. Left alone, a machine might only tell us something useful about facial expressions a fraction of the time. However, there’s a crucial element to this – there’s always a human in the machine.
Facial expression analysis is a tool in the hands of human experimenters. And just like any tool, it can be used well and it can be used badly. Fortunately, in the context of behavioral research, human experimenters are always forced to make human decisions with this tool.
For example, consider setting up an experiment where you want to judge whether or not an audience likes a short film. You make a hypothesis, recruit participants, record their facial expressions, and analyze the data. The first and last steps necessitate human intervention – requiring the wisdom to create both meaningful assumptions and conclusions regarding the data. It’s necessary for the researcher to set up appropriate contexts by which to judge emotions and to interrogate the data in light of this.
Below, we’ll explain what emotions really are, how facial expression analysis can capture emotional expressions, and how to evaluate data with the context in mind.
Ultimately, machine intelligence is only as smart as the humans using it. Using context, human judgment, and the power of artificial intelligence can help us get closer than ever before to accurately judging human emotion at scale.
What are emotions?
While there are differing schools of thought with regards to how an emotion is produced, and the universality of experienced emotions, one of the most dominant lines of thinking today comes from the work of Lisa Feldman Barret. While we can’t do justice to her work within the scope of a blog post, there are two relevant points that are worth discussing further here. These can be boiled down to: 1. emotions are contextual, and 2. there is no single place where emotions reside in the brain (we would recommend further reading of Feldman Barret’s work through the following articles and her book: [1, 2, 3]).
First up: emotions are contextual. This encompasses not only the feeling of emotions, but also how they’re exhibited. As an example, let’s imagine you’re taking a difficult test. There is a range of contexts that can surround this – perhaps you’ve been dreading the test, but you find that you know all the answers, maybe it’s as bad as you feared, or perhaps you’re a person that relishes a good challenge – or maybe you don’t really feel much at all. You could experience different internal emotions that can range the whole gamut from negative to positive. And that’s just what’s going on inside.
On the outside, you could appear to be scowling with anger as you merely concentrate, wide-eyed as you try to take in the information, or maybe you’re the kind of person that’s prone to anxious laughter. The point is this – our internal emotions and our external expressions are both variably shaped by our past, our experiences in the present, and our expectations – our predictions – of the future. Context rules, both inside and out.
Secondly – the location of emotions in the brain, where are they? The answer often ends up pointing to a particular place: everywhere. Meta-analyses (that is, analysis of results from many scientific investigations) have shown that there is very little consistency in the areas of the brain that light up when people say they are experiencing a certain emotion.
For example, the amygdala has been repeatedly pointed to as the seat of the emotion fear [4]. However, research has also shown that activity in the amygdala increases when people experience anything regarded as emotionally “impactful”, valuable, or even “subjectively arousing” [5]. Mapping an emotion to an area of the brain (or vice versa) leads to a variety of results. This has lead Feldman Barret to conclude similarly to what William James suggested over 100 years ago [6] – that the experience of emotions isn’t generated by the brain in a one-to-one manner, but is deduced by the brain from the physiological experience of a situation.
Consider the following: you’re in a doctor’s waiting room, full of adrenaline (in the case of the study that this refers to, quite literally – participants were injected with epinephrine at the start of the study [7]). Another patient comes in, who appears agitated and angry. You’re soon given a questionnaire that asks about your mood. As a result, you’d be more likely to also rate your mood as angry, compared to people who weren’t given an adrenaline boost. The same holds true if you witness another “patient” (actually an actor in both cases) appearing joyful while under the influence of increased arousal. Emotions are readily attributable to our inner state.
What facial expression analysis tells us about emotions
So, to expand on our original question – if there is so much variability in how we create, experience, and express emotions, how can the face tell us anything about how we feel? Context is key.
Exploring emotional responses to stimuli simply requires that the experiment is well-planned and well-executed (as all experiments should be). As the production and creation of emotions is heavily context-dependent, it falls on the researcher to define and understand the context in which the experiment is taking place.
Consider once again an experiment in which you’re testing a short film. You need to build your hypothesis in relation to what you are testing. If you want to see if the film makes people happy, you might test for an increased amount of smiles, relative to a control group who are watching neutral content. You also need to know that the content can reasonably elicit smiles and/or happiness – otherwise, you might be misattributing the expressed emotion. You can then quantify the difference in how people react, although the experiment isn’t over yet.
The analysis stage brings its own challenges (it always does). You will need to examine the data for further potential misclassifications, and to test for significant differences on a group level. For the first part, it’s helpful to explore any unexpected occurrences in the data. For example, does it appear that participants were showing a lot of frustration at a certain point in the film? Perhaps it is frustration, but it could also be a sign of something else, such as increased concentration. This is where human judgment can be essential – to discern how the basic units of facial expressions relate to the stimulus.
Going further with the data
You need to look closely at what the data is telling you. This extends to the second source of variability in emotion production – that emotions don’t reside in a particular place in the brain, yet are perceived through physiological activity in response to the environment around us. By using multiple biosensors in addition to facial expression analysis, we can build a clearer picture of both the inner and outer mechanisms that produce emotions.
Understanding the state of physiological arousal provides another contextual cue by which to judge the data, and brings us closer to really understanding what’s going on.
References
[1] Barrett L. F. (2006). Solving the emotion paradox: categorization and the experience of emotion. Personality and social psychology review, 10(1), 20–46. https://doi.org/10.1207/s15327957pspr1001_2
[2] Lisa Feldman Barrett, James Gross, Tamlin Conner Christensen & Michael Benvenuto (2001) Knowing what you’re feeling and knowing what to do about it: Mapping the relation between emotion differentiation and emotion regulation, Cognition and Emotion, 15:6, 713-724, DOI: 10.1080/02699930143000239
[3] Le Mau, T., Hoemann, K., Lyons, S.H. et al. Professional actors demonstrate variability, not stereotypical expressions, when portraying emotional states in photographs. Nat Commun 12, 5037 (2021). https://doi.org/10.1038/s41467-021-25352-6
[4] Adolphs, R., Tranel, D., Damasio, H., & Damasio, A. R. (1995). Fear and the human amygdala. The Journal of neuroscience, 15(9), 5879–5891. https://doi.org/10.1523/JNEUROSCI.15-09-05879.1995
[5] Lindquist, K. A., Wager, T. D., Kober, H., Bliss-Moreau, E., & Barrett, L. F. (2012). The brain basis of emotion: a meta-analytic review. The Behavioral and brain sciences, 35(3), 121–143. https://doi.org/10.1017/S0140525X11000446
[6] James W. Principles of Psychology. University of Chicago Press; 1890/1998.
[7] Schacter, S., & Singer, J. E. (1962). Cognitive, social, and physiological determinants of emotional state. Psychological review, 69, 379–399. https://doi.org/10.1037/h0046234
