VR technology could enhance remote medical consults

19 September 2023

Canterbury researchers at UC’s HIT Lab NZ are exploring how virtual reality could be used as a tool during remote medical assessments.

  • Rob Lindeman and Dilshani Kumarapeli

    Professor Rob Lindeman and PhD student Dilshani Kumarapeli are part of a research team exploring how virtual reality could be used as a tool during remote medical assessments.

Sustainable Development Goal (SDG) 3 - Good health and wellbeing.

In a recent study PhD student Dilshani Kumarapeli, Postdoctoral Fellow Dr Sungchul Jung, Research Associate Dr Yuanjie Wu, and Professor Rob Lindeman, Director of the Human Interface Technology (HIT) Lab at Te Whare Wānanga o Waitaha | University of Canterbury (UC), investigated the potential for Virtual Reality (VR) headsets in medical consultations where access or risk factors may prevent doctor and patient from being in the same room.

The team’s research was designed to provide the patient and doctor with a personalised experience via VR headset. Kumarapeli and Professor Lindeman call this an “asymmetric system” and see it as an important point of difference when compared with more common applications for VR technology — like video games — where both users employ the same system and receive identical information. 

The researchers believe the highly customised nature of their system could enhance the connection between doctor and patient, even when they’re not in the same room, by providing a sensory experience designed to best address their individual needs. The system could be used for patients based in remote locations, patients with highly contagious illnesses, or patients with mental health conditions that may cause them to behave violently towards others.

“The patient needs to feel trust in their clinician, so for them, the system is focused on relaying eye-contact and facial expressions that make them feel connected to the doctor despite being in a different physical location,” Professor Lindeman says. “For the clinician, it can provide detailed physiological information that the patient might struggle to convey verbally during a remote assessment.”

Data conveyed to the doctor via the headset could include eye-tracking, facial expressions, pulse and breathing rate. This data would also be interpreted by a connected software programme, enabling the clinician to make a more accurate diagnosis.

While the research team has trialled the system with a group of UC students, and both Professor Lindeman and Kumarapeli see great potential to use similar “asymmetric” VR technology in a diverse range of scenarios, including educational frameworks, the system will be further developed for use in real-world scenarios.

For Kumarapeli, the most rewarding aspect of the research was developing the emotion recognition software. 

“Because the participants are wearing the headset, we weren’t able to capture all their facial expressions with a camera, so we had to find an alternative to solve this issue,” she says.

“We used a facial capture device and trained a neural network to identify various key emotions. These emotions were then relayed to the clinician, to help in managing the diagnosis session. I really enjoyed the journey of developing the neural network and applying the trained modules to this real-life scenario.”

The researchers received support from the Science for Technological Innovation National Science Challenge. Their work is being prepared for publication.

Dr Jung was the principal investigator on this work, and has since taken a continuing Assistant Professor position in the United States.

Media contact:

  • Email: media@canterbury.ac.nz Ph: (03) 369 3631 or 027 503 0168

University of Canterbury wins 2023 Tertiary Championship, secures historic ...

In a historic win, the University of Canterbury team emerged as victors of the 2023 National Tertiary Championship Series, marking a milestone in ...

Amy Adams and Cheryl

UC Vice-Chancellor speaks at Cambridge University

Invited to speak at Cambridge University, UC’s Tumu Whakarae | Vice-Chancellor highlighted the historic ties between the universities in front of an ...