The field of facial expression analysis is over a hundred years old, and has now come of age. The detection of expressions and emotions by automatic analysis has matured into a reliable methodology that is widely used in a variety of research. The advanced methods that we now see have of course depended on the work previously carried out.
In order to understand the current science of facial expression analysis, and to see where the field is going, it can help to see where it’s come from. We’ve put together some of the highest ranking articles concerned with facial expression analysis, chosen by number of citations, and with iMotions’ expertise (including our own FACS-certified Thomas Terkildsen).
So if you’re looking to brush up your knowledge, get up to speed, or want to check your expertise, have a read through our top 5 research articles. All of which should help you in your understanding of what has been – and will continue to be – important in the facial expression analysis world.
Autonomic Nervous System Activity Distinguishes Among Emotions
Ekman, Levenson, and Friesen, 1983 (2683 citations)
Ekman – essentially the father of facial expression analysis – is inevitably the prime feature of this list. By creating the Facial Action Coding System (FACS) book, Paul Ekman and Wallace Friesen put forth their comprehensive categorization system of action units of the face – discrete movements that can culminate in entire expressions.
It is difficult to overstate the importance of that work, as it has shaped the entirety of the facial expression analysis field. That body of work, while critical, was not the only piece of scientific work from Ekman and the above article is a good example of one his many other contributions.
Ekman, Levenson, and Friesen showed how exhibiting a facial expression can lead to changes in autonomic activity. By measuring skin temperature, galvanic skin response (GSR), and electrocardiography (ECG) of participants who were guided into making exact facial expressions, the researchers found that certain expressions led to significant physiological changes.
For example, performing a facial expression related to anger would lead to an increase in the temperature of the extremities (such as the fingers). Ultimately, the researchers were able to distinguish the physiological responses related to expressions of anger, sadness, fear, and disgust – showing that we not only react to the emotions displayed by others, but also the emotions displayed by ourselves.
The Expression of the Emotions in Man and Animals
Charles Darwin, 1872 (14206 citations)
No list of literature concerning facial expression analysis would be complete without mentioning Darwin’s work on human expression. Through ongoing discussions with psychologists of the time, Darwin posits that the formation of certain traits related to facial expressions occurs through genetic inheritance.
Darwin’s work was particularly important for his investigations (and ultimate conclusion) that facial expressions are universal (a topic that has been supported since in numerous studies, yet is still debated).
By establishing the common thread between people, Darwin laid the groundwork for the systematic evaluation of facial expressions that would soon follow.
Etcoff and Magee, 1992 (554 citations)
This article by Etcoff and Magee (1992), was pivotal in the finding that the recognition of a facial expression is likely done by discrete categorization (e.g. a face is either happy or not). This suggests that our perception of facial expressions works in much the same way that our recognition of shades of colors works – with identifiable boundaries eliciting a set categorization.
This is important as the definition of facial expressions could either exist as an impression along a continuum, or as discrete variables. If our perception is more keen to make concrete categorizations, it means that uncertainty with facial expressions isn’t beneficial when it comes to understanding the emotions of other people.
This study was carried out using computer generated images, but was later supported by a study using warped photographs – a scenario more closely fitting real life.
The neuropsychology of facial expression: a review of the neurological and psychological mechanisms for producing facial expressions.
Rinn, 1984 (723 citations)
For a clear introduction of the field of facial expressions – and a summary of where the field was before the FACS was introduced – Rinn’s review makes for informative reading. Despite being published over thirty years ago, the article provides a comprehensive rundown of the elements of facial expressions, and their analysis.
Written in an accessible format, the article is a particularly useful primer for understanding the facial muscles, nerves, and even the bones involved in generating facial expressions. The text also serves well as a reminder of how far the field has come. Now that facial expressions can largely be assessed and quantified automatically, it is impressive to think of the time that must have been involved in “reviewing the tape twice, once to look for activity in the upper face, and once for the lower face”.
The article also features useful demonstrations of the facial nerves, and how they can be impacted by neurological damage. Paralysis of facial nerves can have different effects, depending on the origins of the damage – examples of this are shown with pictures (such as with a 61 year-old woman showing involuntary non-emotional laughing).
By using grounded and accessible scientific explanations of facial expressions, Rinn provides a holistic understanding of the essential features of the facial expression analysis field.
DEAP: A Database for Emotion Analysis; Using Physiological Signals
Koelstra, Soleymani, Lee, Yazdani, Ebrahimi, Pun, Nijholt, and Patras, 2011 (540 citations)
This article by Koelstra and others (2011) provides insight into not only where the field is at, but also where the field of facial expression analysis is heading. The use of various biometric (physiological) signals to determine the emotional state of participants is an approach that is increasingly used to provide deeper information about the participants, and to increase the strength of the data.
The article also provides the data as a freely accessible database, that can be used for further studies, and validation of current data sets. While designed as a way in which to help design a music video recommendation system, the data taps into a deeper insight – that multimodal research can help understand affective responses in a reliable and flexible way.
By using ECG, GSR, and EMG, the researchers construct a multidimensional picture of each participant’s affective response, finding that the use of multiple sensors provides a corroborated and holistic view of the participant. This is something we will surely see more of, in the facial expression analysis world.
I hope you’ve enjoyed reading our list of the top 5 facial expression research articles. If you’d like to learn more about facial expression analysis – with more findings from the field – then download our free guide below.