The field of facial expression analysis (FEA) is over a hundred years old, and has now come of age. The detection of emotional expressions by automatic analysis has matured into a reliable methodology that is widely used in a variety of research. These advanced methods have of course stood on the shoulders of previous research.
In order to understand the current science of facial expression analysis, and to see where the field is going, it can help to see where it’s come from. We’ve put together some of the highest ranking articles concerned with facial expression analysis, chosen by number of citations, and with iMotions’ expertise.
So if you’re looking to brush up your knowledge, get up to speed, or want to check what you know, have a read through our top 10 research articles. All of which should help you in your understanding of what has been – and will continue to be – important in the facial expression analysis world.
Autonomic Nervous System Activity Distinguishes Among Emotions
Ekman, Levenson, and Friesen, 1983 (2683 citations)
Ekman – essentially the father of facial expression analysis – is inevitably the prime feature of this list. By creating the Facial Action Coding System (FACS) book, Paul Ekman and Wallace Friesen put forth their comprehensive categorization system of action units of the face – discrete movements that can culminate in entire expressions.
It is difficult to overstate the importance of that work, as it has shaped the entirety of the facial expression analysis field. That body of work, while critical, was not the only piece of scientific work from Ekman and the above article is a good example of one his many other contributions.
Ekman, Levenson, and Friesen showed how exhibiting a facial expression can lead to changes in autonomic activity. By measuring skin temperature, galvanic skin response (GSR), and electrocardiography (ECG) of participants who were guided into making exact facial expressions, the researchers found that certain expressions led to significant physiological changes.
For example, performing a facial expression related to anger would lead to an increase in the temperature of the extremities (such as the fingers). Ultimately, the researchers were able to distinguish the physiological responses related to expressions of anger, sadness, fear, and disgust – showing that we not only react to the emotions displayed by others, but also the emotions displayed by ourselves.
Check out our recent publications made possible by FEA & iMotions
The Expression of the Emotions in Man and Animals
Charles Darwin, 1872 (14206 citations)
No list of literature concerning facial expression analysis would be complete without mentioning Darwin’s work on human expression. Through ongoing discussions with psychologists of the time, Darwin posits that the formation of certain traits related to facial expressions occurs through genetic inheritance.
Darwin’s work was particularly important for his investigations (and ultimate conclusion) that facial expressions are universal (a topic that has been supported since in numerous studies, yet is still debated).
By establishing the common thread between people, Darwin laid the groundwork for the systematic evaluation of facial expressions that would soon follow.
Categorical perception of facial expressions
Etcoff and Magee, 1992 (554 citations)
This article by Etcoff and Magee (1992), was pivotal in the finding that the recognition of a facial expression is likely done by discrete categorization (e.g. a face is either happy or not). This suggests that our perception of facial expressions works in much the same way that our recognition of shades of colors works – with identifiable boundaries eliciting a set categorization.
This is important as the definition of facial expressions could either exist as an impression along a continuum, or as discrete variables. If our perception is more keen to make concrete categorizations, it means that uncertainty with facial expressions isn’t beneficial when it comes to understanding the emotions of other people.
This study was carried out using computer generated images, but was later supported by a study using warped photographs – a scenario more closely fitting real life.
The neuropsychology of facial expression: a review of the neurological and psychological mechanisms for producing facial expressions
Rinn, 1984 (723 citations)
For a clear introduction of the field of facial expressions – and a summary of where the field was before the FACS was introduced – Rinn’s review makes for informative reading. Despite being published over thirty years ago, the article provides a comprehensive rundown of the elements of facial expressions and their analysis.
Written in an accessible format, the article is a particularly useful primer for understanding the facial muscles, nerves, and even the bones involved in generating facial expressions. The text also serves as a reminder of how far the field has come. Now that facial expressions can largely be assessed and quantified automatically, it is impressive to think of the time that must have been involved in “reviewing the tape twice, once to look for activity in the upper face, and once for the lower face”.
The article also features useful demonstrations of the facial nerves, and how they can be impacted by neurological damage. Paralysis of facial nerves can have different effects, depending on the origins of the damage – examples of this are shown with pictures (such as with a 61 year-old woman showing involuntary non-emotional laughing).
By using grounded and accessible scientific explanations of facial expressions, Rinn provides a holistic understanding of the essential features of the facial expression analysis field.
DEAP: A Database for Emotion Analysis; Using Physiological Signals
Koelstra, Soleymani, Lee, Yazdani, Ebrahimi, Pun, Nijholt, and Patras, 2011 (540 citations)
This article by Koelstra and others (2011) provides insight into not only where the field is at, but also where the field of facial expression analysis is heading. The use of various biosensor (physiological) signals to determine the emotional state of participants is an approach that is increasingly used to provide deeper information about the participants, and to increase the strength of the data.
The article also provides the data as a freely accessible database, that can be used for further studies, and validation of current data sets. While designed as a way in which to help design a music video recommendation system, the data taps into a deeper insight – that multimodal research can help understand affective responses in a reliable and flexible way.
By using ECG, GSR, and EMG, the researchers construct a multidimensional picture of each participant’s affective response, finding that the use of multiple sensors provides a corroborated and holistic view of the participant. This is something we will surely see more of, in the facial expression analysis world.
Automatic Facial Expression Analysis: A Survey
Fasel & Luettin, 2003 (2207 citations)
In 2003, Beat Fasel and Juergen Luettin provided a walkthrough of progress that had been made in the field of facial expression analysis in the decade prior. While the article approaches the topic through the lens of computer vision / engineering, the authors created an important and thorough primer for future analysis frameworks to build from.
The article may be of limited applicability for those interested purely in the emotional or cognitive elements of facial expressions, yet the article is a helpful reference when learning more about the algorithms and approaches underlying most modern software-based facial expression analysis methods.
Unconscious Facial Reactions to Emotional Facial Expressions
Dimberg, Thunberg, & Elmehed, 2003 (2022 citations)
Researchers from Uppsala University, Sweden, showed that people will mirror facial expressions of other people, even if they are unaware of seeing any particular expression. Participants were made to observe images of facial expressions (from Ekman and Friesen’s Pictures of Facial Affect, 1976), while an fEMG device recorded the minute facial muscle movements (specifically on the zygomaticus major and the corrugator supercilii muscles). Images of emotional expressions (e.g. angry, happy) were shown for 30ms – below the threshold of conscious perception. This was also immediately followed by presentation of a neutral facial expression.
It was found that activity increased within facial muscles that corresponded to the nonconsciously presented facial expression, yet none of the participants were able to accurately report the emotion of the prior exposure, suggesting that emotions are, or at least can be, processed by a separate nonconscious system.
This article was pivotal in showing that reactions to emotional expressions can be automatic and elicited even without conscious awareness – knowledge that is crucial in understanding how emotions can be shaped by interactions and the environment an individual is in.
A Specific Neural Substrate for Perceiving Facial Expressions of Disgust
Phillips, Young, Senior, Brammer, Andrew, Calder, Bullmore, Perrett, Rowland, Williams, Gray & David, 1997 (1948 citations)
Part of understanding facial expressions involves understanding not only how or why they are produced, but also how they impact other people. By using fMRI to examine brain activity while participants viewed images of “disgust” facial expressions, the researchers were able to show that facial expressions of disgust evoked a distinct part of the brain than images of other facial expressions.
The activation was found specifically within the anterior insular cortex (roughly located in the middle of your brain, near the front), and areas linked to the limbic cortico–striatal–thalamic circuit. Activation of the insular cortex is of particular interest as gustatory disgust responses (for example, seeing rotten food) are also known to increase activity within this area. This shows that there is an overlap in both our disgust response, and our response to other people’s disgust responses. The authors suggest that part of our disgust response may be shaped through learning from the responses of other people – meaning that our feelings of disgust could be socially created.
FaciaI Expressions are Contagious
Lundqvist & Dimberg, 1995 (411 citations)
The emotional contagion hypothesis suggests that emotions are shaped by how people interact – if I show an expression of happiness, you are likely to emulate and experience that emotion, and this can then spread further to others. Aiming to test this hypothesis, researchers used fEMG to track the activity of several facial muscles (the zygomaticus major, the levator labii, the frontal lateralis, and the corrugator supercilii) while participants viewed images of various different facial expressions.
The results showed that the muscle movements were broadly correlated with the expression that the participants witnessed (although with the intriguing exception that responses to the fear expression did not meet statistical significance). The participants also completed a survey that aimed to measure their inner emotional responses – the results suggested that the participants also experienced an increase in the emotion that matched the image.
While the experiment was not able to explain the mechanisms that could lead to the responses (i.e. if the response was just mimickry, or an actual emotional response), the research still added to an understanding of how facial expressions can impact others.
Voluntary Smiling Changes Regional Brain Activity
Ekman & Davidson, 1993 (396 citations)
A widely debated issue within studies of facial expressions and emotions is whether or not a change in the former can bring about a change in the latter – if someone smiles, do their emotions change too, even when the smile is voluntary? Ekman and Davidson set out to answer this question with the use of EEG (it should be noted that this is still a contentious topic, and this study was, while important, not definitive). The researchers compared the level of brain activity elicited while participants exhibited a “Duchenne smile” (a genuine smile), or after they were told “part your lips and let your lip corners come up” (a non-genuine smile).
The researchers reported differences in brain activity across these smiles – taken as evidence that differences in emotional experiences can occur as a result of the way in which an individual smiles. While the debate continues around the significance of such findings (and in other published studies in subsequent years), the research suggests that an approach using multiple data streams can provide a greater understanding of the processes involved in generating facial expressions.
I hope you’ve enjoyed reading our list of the top 10 facial expression research articles. If you’d like to learn more about facial expression analysis – with more findings from the field – then download our free guide below.
