VR: Training and Performance [Part 1]

The current generation of Virtual Reality (VR) devices provide the possibility to experience new worlds with unprecedented fidelity. Immersion in environments that are impractical, dangerous, or even impossible to encounter can now be carried out with the help of the devices that are often no larger than a pair of binoculars.

VR in its current form, appearing from around 2010, is now also more accessible than ever – many devices that appeared before were simply too expensive for the average consumer (and certainly didn’t offer the level of detail and engagement that is possible with modern devices).

As VR is now much more available, the potential of these devices is starting to be realized. While principal development was focused on improving VR for gaming purposes, the possibilities for improving lives is much more broader than this, and governmental institutes, universities, and companies are seeking to utilize VR to its fullest potential.

Among the possibilities offered by VR, therapy (which we’ve covered here before), design and architecture, and training are at the top of the list of most researched and worked with domains (as well as offering great potential to improve health and well-being).

Training in VR benefits from the core advantage of any virtual environment – that it can be created as desired, including environments that would be unreasonable or unethical to put people in if the situation was real.

Below, we will go through the most popular and well-studied use cases for VR – medical, physical, social, military, industry training – and discuss current research into their use. This post is the first in a two part series and covers the first three topics – find part two here through this link.

Virtual Reality Medical / Surgery Training

Even relatively early on in the use of VR was it already apparent that medical training within virtual environments could not only be beneficial as it bore no risk to patients, but also that it could help surgical residents improve as an adjunct to their training.

A study by Seymour and others from Yale University and Queens University in Belfast found that not only was dissection of a gallbladder 29% faster on average with those trained additionally in a VR environment (as compared to those who only underwent standard training), but that they were also much less likely to injure, or burn non-target tissue [1].

Research in 2009 found similar results with laparoscopic surgery (surgery within the abdomen or pelvis). Quite drastically, the researchers within Denmark found that prior VR training improved average performance within actual surgery to be equivalent to carrying out 20-50 prior operations, whereas the control group performance was rated to be equivalent to fewer than 5 previous operations [2].

A meta-analysis carried out in 2016 confirmed these findings for laparoscopic surgery, analyzing the results from 579 participants who took part in either VR or control conditions [3]. The researchers found that overall significant improvements in surgery performance occurred for those in the VR training conditions.

While the above studies are limited to particular types of surgeries, they still present an important proof-of-concept – that real world skills can be improved from practice within the virtual world.

A recent study in 2019 showed this once again with hip replacement surgery [4]. A similar experimental design (with surgery performance rated by other surgeons who were blind to the experimental conditions) yielded similar results. As the authors of the study stated, the “data demonstrate transfer of procedural knowledge and psychomotor skills learnt from VR to a real-world setting… VR training advanced them further up the learning curve”.

The implications of being able to improve surgical skills drastically by training within VR are difficult to overstate – if the advantages are shown to be consistent across other forms of surgical treatment, an overhaul of operating room education (and better surgery results) could well be on the way.

As a 2016 article published in the Annals of Translational Medicine stated “As dedicated, structured curricula are developed that incorporate simulations into daily resident training, simulated surgeries will strengthen the surgeon’s skill set, decrease hospital costs, and improve patient outcomes” [5].

Virtual Reality for Sports and Physical Therapy Training

In line with VR surgery training, physical therapy and sports training also seek to capitalize on the motor skill improvements available in simulated environments.

This has been applied to learning strategy for basketball [6], American football [7], and tennis [8], among other sports. Software has been developed allowing the gameplay to be entered prior to player use, allowing the athlete to practice set pieces before playing in real life.

This has the clear advantage of allowing players to make mistakes and rigorously prepare before entering into a game where the stakes are higher. Combined with tools such as eye tracking, it’s possible to identify where the player’s visual attention lies, and to see if this impacts performance (and then offer solutions to improve gameplay).

Physical therapy also offers the opportunity to practice movements within environments that are safe to make errors within. Researchers have investigated the use of VR environments in balance and gait training with elderly people [9]. The use of games allows individuals to both train their movements in settings that are more rewarding than ordinary exercise (for example, in a skiing game), but without the risk that would be provided by such environments (such as with actual skiing).

Physical therapy is particularly important for recovery post-stroke, as the beneficial effects are dose-dependent – the more someone is able to practice their movements, the better they will be able to recover [10]. Barriers to this are many, but chief among them are logistics and cost – not everyone can access physical therapy easily, and not everyone can afford it.

Virtual reality at least helps to reduce those barriers – while the headsets aren’t exactly cheap, they are at least relatively much cheaper than therapy, and come with the additional benefit that they can be used as often as desired in one’s own home.

Wittman and other researchers from University Hospital Zurich investigated the effects of using VR with patients who had experienced a stroke [11]. They found that the VR system was used to a high degree by patients, and that ultimately motor functions improved with increased use, suggesting that this system could work well as an addition or partial replacement for physical therapy in a clinic or hospital.

Research has also shown similar improvements with children with cerebral palsy. Researchers at several universities in Korea found that VR environments facilitate greater improvement in motor function as compared to non-VR training [12].

The use of EMG devices can provide more data about the increase in function, by quantifying the actual change in muscle activity. Other devices such as EEG can provide crucial information about neural activity that can be associated with changes leading to motoric improvements or rehabilitation.

Social Training For Autism

Another area in which VR is being used to improve function is in the training of social skills for individuals with autism. Autism is often characterized by “Difficulty with communication and interaction with other people”, according to the DSM-5 (Diagnostic and Statistical Manual of Mental Disorders, [13]). VR offers the opportunity for these skills to be trained, in the same manner that physical therapy can be carried out.

Researchers from the University of Texas administered VR training to children with high-functioning autism between the ages of 7 and 16 [14]. The children entered the VR platform Second Life™, and experienced various social settings with a virtual avatar of a clinician. Various practice sessions were encountered, such as “small-talk in conversation at the school yard” or “actively participate in a group in the classroom”. The children were also coached by a present clinician to help them navigate each scenario.

As the authors state: “Each social scenario was designed to emphasize a targeted social learning objective in varying contexts, such as meeting new people, dealing with a bully, bonding with friends, confronting conflict, consoling a friend, or handling social dilemmas (i.e., meeting a stranger, catching someone cheating)”. Following this, the children completed measures of skills with emotion recognition, social attribution, attention and executive function using various tests.

The researchers found that the emotion recognition, social attribution, and partially the executive function capabilities of the children in the experiment improved following several weeks of continued practice.

There were however some limitations with the study, as the authors state: “future research using VR to train social cognition may benefit from facial tracking of emotions” – suggesting that facial expression analysis (with VR headsets, this would have to be done with facial electromyography) could have a part to play in testing emotion recognition and training. The authors also point out that eye tracking within VR could be of use, stating how “A recent investigation by Barisic et al. (2013) showed the feasibility and effectiveness of using a dual eye tracking system in real time social interactions” [15].

The use of eye tracking with VR can help automate the collection of data concerning attention, showing the visual areas that were seen more or less than control groups. This reduces the resources needed to assess attentional processes (such as with assessment sets), and also allows the processes to be measured directly, and in real time.

Other research has built upon the same principles, yielding improvements in social skills for individuals with autism in job interview contexts [16], teaching road safety [17], and driving skills [18], among other studies.

Researchers from Vanderbilt University used a variety of measures to assess the performance of autistic individuals in a virtual reality driving environment [18]. As the authors state: “Signals of interest included electromyogram (EMG), electrocardiogram (ECG), galvanic skin response (GSR), photoplethysmogram (PPG), skin temperature, and respiration. These signals were selected because of their ability to predict the affective state of the subject [Cacioppo et al. 2007].” [19].

The researchers also used an Emotiv EEG device to collect data surrounding cognitive states, as well as “rotational velocity information” that was used to assess driving performance. Most central to the study was the use of eye tracking to follow the participants’ gaze patterns.

By using an adaptive gaze-contingent paradigm, the researchers were able to construct a VR driving environment that responded to where the participant was looking, “with the aim of providing a more individualized method of driving intervention”.

The multimodal approach shows great promise when implemented alongside VR environments, allowing any setting to be investigated at depth.

Conclusion

VR in its current incarnation is still a relatively new technology, yet still shows great promise in being able to provide novel environments for novel testing. The potential of virtual training for different scenarios or with different populations is vast, and provides benefits such as reduced cost, reduced stress, increased motivation, as well as making some settings simply accessible for training.

Each application area should of course be thoroughly tested to support or refute the possible benefits, and biosensors are some of the tools that are able to provide the data for that. Early data however does look promising, pointing the way towards better training and performance beyond virtual environments.

This was the first of our articles looking at training and performance in VR. The second article is available here through this link.

References

[1] Seymour, N., Gallagher, A., Roman, S., O’Brien, M., Bansal, V., Andersen, D. and Satava, R. (2002). Virtual Reality Training Improves Operating Room Performance. Annals of Surgery, 236(4), pp.458-464.

[2] Larsen, C., Soerensen, J., Grantcharov, T., Dalsgaard, T., Schouenborg, L., Ottosen, C., Schroeder, T. and Ottesen, B. (2009). Effect of virtual reality training on laparoscopic surgery: randomised controlled trial. BMJ, 338(may14 2), pp.b1802-b1802.

[3] Alaker, M., Wynn, G. and Arulampalam, T. (2016). Virtual reality training in laparoscopic surgery: A systematic review & meta-analysis. International Journal of Surgery, 29, pp.85-94.

[4] Hooper, J., Tsiridis, E., Feng, J., Schwarzkopf, R., Waren, D., Long, W., Poultsides, L., Macaulay, W., Papagiannakis, G., Kenanidis, E., Rodriguez, E., Slover, J., Egol, K., Phillips, D., Friedlander, S. and Collins, M. (2019). Virtual Reality Simulation Facilitates Resident Training in Total Hip Arthroplasty: A Randomized Controlled Trial. The Journal of Arthroplasty.

[5] Badash, I., Burtt, K., Solorzano, C. and Carey, J. (2016). Innovations in surgery simulation: a review of past, current and future techniques. Annals of Translational Medicine, 4(23), pp.453-453.

[6] Tsai, W., Chung, M., Pan, T. and Hu, M. (2017). Train in Virtual Court: Basketball Tactic Training via Virtual Reality. In: MultiEdTech ’17.

[7] Huang, Y., Churches, L., & Reilly, B. (2015). A case study on virtual reality American football training. Paper presented at the ACM virtual reality international conference (VRIC), Laval, France.

[8] Kawamura, S.; Ida, M.; Wada, T.; Wu, J.L. Development of a virtual sports machine using a wire drive system—a trial of virtual tennis. In Proceedings of IEEE/RSJ International Conference Intelligence Robots and System, Pittsburgh, PA, USA, 1995; pp. 111-116.

[9] de Vries, A., Faber, G., Jonkers, I., Van Dieen, J. and Verschueren, S. (2018). Virtual reality balance training for elderly: Similar skiing games elicit different challenges in balance training. Gait & Posture, 59, pp.111-116.

[10] Kwakkel, G. (2006). Impact of intensity of practice after stroke: issues for consideration. Disabil Rehabil, 28(13–14):823–30.

[11] Wittmann, F., Held, J., Lambercy, O., Starkey, M., Curt, A., Höver, R., Gassert, R., Luft, A. and Gonzenbach, R. (2016). Self-directed arm therapy at home after stroke with a sensor-based virtual reality training system. Journal of NeuroEngineering and Rehabilitation, 13(1).

[12] Cho, C., Hwang, W., Hwang, S. and Chung, Y. (2016). Treadmill Training with Virtual Reality Improves Gait, Balance, and Muscle Strength in Children with Cerebral Palsy. The Tohoku Journal of Experimental Medicine, 238(3), pp.213-218.

[13] American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (5th ed.). Arlington, VA.

[14] Didehbani, N., Allen, T., Kandalaft, M., Krawczyk, D. and Chapman, S. (2016). Virtual Reality Social Cognition Training for children with high functioning autism. Computers in Human Behavior, 62, pp.703-711.

[15] Barisic, I., Timmermans, B., Pfeiffer, U.J., Vogeley, K., Schilbach, L., 2013. Using dual eyetracking to investigate real-time social interactions, in: Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems. Presented at the ACM SIGCHI
Conference on Human Factors in Computing Systems, Paris.

[16] Smith, M., Fleming, M., Wright, M., Losh, M., Humm, L., Olsen, D. and Bell, M. (2015). Brief Report: Vocational Outcomes for Young Adults with Autism Spectrum Disorders at Six Months After Virtual Reality Job Interview Training. Journal of Autism and Developmental Disorders, 45(10), pp.3364-3369.

[17] Strickland, D., Mesibov, G.B., & Hogan, K. (1996). Two case studies using virtual reality as a learning tool for autistic children. Journal of Autism & Developmental Disorders 26:651–659.

[18] Zhang L., Wade J., Bian D., Fan J., Swanson A., Weitlauf A., et al. (2017). Cognitive load measurement in a virtual reality-based driving system for autism intervention. IEEE Trans. Affect. Comput. 8 176–189. 10.1109/TAFFC.2016.2582490.

[19] Cacioppo, J. T., Tassinary, L. G., and Berntson, G. (2007). Handbook of Psychophysiology. Cambridge University Press, Cambridge.