At iMotions, we’re great believers in making the complex achievable and understandable – this goes not only for our research platform, but also for this blog. Recently, we have seen several high impact articles published, using iMotions to take on complex problems.

We want to show you the ins and outs of these articles, so you can see how iMotions works out in the field. We have chosen to focus on a new innovative piece of research that we feel exemplifies the iMotions approach to understanding human behavior.

Read on to learn about the groundbreaking new research being carried out.


The study we’re looking at is concerned with Autism Spectrum Disorder (ASD). ASD is a multifaceted neurodevelopmental disorder – it consists of impaired social and communication skills, and constrained or repetitive behaviors.  It affects about 1% of the population worldwide and is wide-ranging in severity across individuals.

Despite the impact in can place upon individuals and families, therapy remains the only approach that can positively affect people with ASD. One of the core goals of autism research is to better understand the disorder, by following treatment more closely, and examining the prevalence of ASD subtypes.

The Study

This challenge was taken on by Seth Ness from Janssen Research and Development (owned by Johnson and Johnson), along with colleagues from Duke University, Northeastern University, and the University of California (among others).

university research autism groups

They designed the JAKE (Janssen Autism Knowledge Engine) system – a set of measurements designed to facilitate further understanding of autism. This study in particular set out to validate the system to help advance knowledge in the future – to assess the feasibility of collecting such a complex set of data.

The study followed 29 ASD children, and 6 typically developing children (both with an average age of 10), using data from a variety of biometric devices, and also from sources such as caregiver journals, medical histories, and activity levels, among others. All of this provided a comprehensive and far-reaching view of the individual’s experience.

Using the JAKE biosensor array, consisted of:

  • Eye tracking
  • EEG
  • Facial expression analysis
  • ECG
  • Galvanic skin response

The researchers were then able to objectively follow the participants both continuously (gathering information on a daily basis), and periodically (at discrete intervals).

multimodal autism research tools

With the help of caregivers, crucial information about the participant’s well-being could be recorded throughout the research period. Discrete tasks were also administered at periodic intervals, which were centered around a specific biometric modality, which are detailed below. Cognitive function was also assessed through the Cogstate Computerized Test Battery, designed to measure ASD-associated cognitive deficits.

Through the use of multiple data streams, the researchers hope to develop biomarkers (objective biological measurements that are associated with a disorder) that can help detect subtypes of ASD, and also provide objective measurements of how therapeutic treatments can impact individuals.

We’ll go through how each of these biometric sensors were used within the study, and what they could help reveal about ASD.

Eye tracking

The eye tracking was collected by a state-of-the-art Tobii X2-30, and used across four different tasks, each designed to provide information about a different aspect of autism. The four tasks were:

eye tracking autism research list

The first video features either a woman showing eye-contact and child-directed speech, or non-social cues (toys). The aim is to see where the child focuses their attention, as previous research has shown that eye-contact and child-directed speech is more likely to keep the attention of typically developing children, than ASD children.

The VET task features a wide variety of images, with various types of content, displayed at the same time – the children are free to look at the scene, and the researchers are able to determine where their preferences lies. Some of the images were previously shown to be of high or low interest to people with ASD, so the amount and type of visual preference can be determined.

The Biological Motion Preference task shows a series of dots that either look like a human walking, or move in a random left-right fashion. The Activity Monitoring task shows acted out scenes, with distracting visual cues in the background. For both of these tasks, ASD children are more likely to pay attention to the non-biological, and visual cues in the background.

eye tracking autism research

Each of these tasks provides a thorough examination of the preferences shown by ASD children. The data can be examined for minor differences, and can inform theories about possible subtypes of autism, and the subtle changes that may occur in response to therapeutic treatments.


EEG was used in a similar manner, with four central visual tasks. These were:

EEG autism research list

The titles of these tasks are a bit more self-explanatory, and the theory around each remains much the same – the tasks that are associated with more social behavior (looking at the eyes, following social scenes closely, etc), are typically less tightly followed by ASD participants.

As with the eye tracking task, each task was performed to allow the researchers to examine potential differences, but in this instance with brain activity (power spectra, asymmetries, and event-related potentials were all collected- explanations of which are a bit beyond the scope of this blog post, but in essence reflect the level of activity in different brain regions, at different times).

eeg autism research

The investigation can then focus on how, when, and where brain activity is increased or decreased in general, and in response to social stimuli. As EEG has such a high temporal resolution (i.e. can collect many datapoints in a short amount of time), it provides a sensitive measure for analysis.


ECG was used to derive a range of measures that pertain to heart rate variability. Combined with other sensors, the ECG recordings could therefore provide information about the level of physiological arousal that the participants were experiencing while completing the tasks above, or other tasks within the study.

Increased heart rate variability has been found to be associated with increased social cognition – typical performance in social environment, suggesting that ASD participants may have a lower heart rate variability than typically developing children.


Due to the portability (and relatively low power consumption) of the GSR device, it was possible to collect GSR data continuously, both within the tasks and in everyday life.

The GSR device provides a great method of examining the physiological arousal of the participants, through recordings of both tonic and phasic GSR activity.

Research has shown that GSR responses of ASD children is not necessarily changed in response to a social stimuli (e.g. a face), which is usually the case with typically developing children. This suggests that GSR changes wouldn’t be expected in the ASD children within this study.

Facial Expression Analysis

The facial expression analysis was carried out through FACET in iMotions, and was recorded when the participants were engaged in any of the tasks mentioned above, but was primarily focused on analyzing the emotional responses to “a set of videos chosen for their humorous visual content”.

Participants were also asked to reproduce specific facial expressions to words that were shown on screen. This meant that the word “happy” would be shown, and the participants would be required to smile in response.

 facial expressions emotion research

Both of these latter tasks are centered around the participant’s ability to exhibit typical emotional responses. As ASD is primarily a disorder of social communication, deficits in this area would be expected, as compared to typically developing children.

The Findings

As the study was focused on the feasibility of carrying out such a complex multi-modal system with a diverse patient group, the results aren’t (yet) based around how the ASD children responded, but are instead taken as a proof-of-concept for such a rigorous experimental design.

The authors conclude that the research approach is not only possible, but will be able to provide much needed information about ASD (they also note that certain aspects will be changed based upon their experiences, although nothing that impacts the approach of the study). They also note that no safety issues were encountered, making this a suitable procedure for studying a vulnerable population.


The authors expect that the research will be able to provide information about biomarkers for ASD, and that ongoing measurements can use these biomarkers to objectively provide information about treatment outcomes.

While the work is far from over, it could well be the beginning of of smart, and personalized treatment for those who need it with ASD, helping improve deficits where they exist.

I hope you’ve enjoyed reading about this innovative new research being carried out with the help of iMotions. To have a look at more groundbreaking research, look through our publication list, or reach out to us to hear about how iMotions is helping researchers worldwide, in universities and companies, to solve some of the biggest questions in human behavior research.

If you’d like to learn more about one of the principal methods in this article – eye tracking – and how it can specifically provide objective information about human behavior, download our free (and recently updated) guide below.

Eye Tracking Pocket Guide Insert