Computer science team uses AR to treat Parkinson’s
Their software and augmented reality headsets allow clinicians to analyze patient movement and guide physical therapy.
There is no known cure for Parkinson’s disease, but studies show that consistent, clinician-guided physical therapy can improve a patient’s range of outcomes and quality of life.
But what happens to a patient whose insurance coverage for physical therapy has run out? Or to rural patients without access to facilities? Or to those struggling to stand or walk at home, with no clinician to guide them?
A UNC-Chapel Hill computer science research team is addressing those issues with a cutting-edge solution: augmented reality.
Since September 2022, Henry Fuchs, Danielle Szafir and others have collaborated with medical researchers at Carolina on the multifaceted Parkinson’s Project. Their PD-Insighter software combines graphics and data visualization to allow clinicians treating Parkinson’s disease to watch and analyze patient movement in real-world home scenarios.
AR headsets and motion sensors track patient movements and interactions with their surroundings to provide the data.
“We could only capture this data with technology that you’re wearing every day,” said Fuchs, the Federico Gil Distinguished Professor of Computer Science. “So what is it that you’ll wear every day 10 years from now? We think it’ll be your glasses, which will have some cameras that could look out, look down and look in, so that they could capture your facial expression and capture your body and so on.”
The metaverse is the “next big leap” in tech, says Fuchs, who believes AR glasses will eventually replace phones the same way phones replaced laptops.
AR glasses differ from virtual reality headsets, like the ones used in gaming, because the wearer can still see the world around them, augmented with graphics. AR headsets let Parkinson’s patients live their daily lives with minimal disruption, while also collecting data for clinicians to study.
However, it’s not enough just to collect data points.
“What do you do when you have all that data?” said Szafir, an assistant professor of computer science. “How do you make it intuitive? How do you help people leverage their own expertise so that you don’t have to be professional data scientists to make sense of all this information that’s flowing in?”
Doctoral research assistant Jade Kandel played a lead role fleshing out the PD-Insighter software and its desktop-based overview dashboard, which labels the patient’s actions throughout the day and highlights “freezes” or motor deficits that clinicians can sort through in minutes instead of hours.
The team also worked on an immersive replay component, which allows clinicians – through their own AR headsets — to replay a patient’s activities. The replay allows clinicians to view a 3D body skeleton of their patients and even see digital recreations of the patient’s environment, such as furniture the patient may use for support.
The more the Parkinson’s Project team develops these tools and collaborates with medical researchers, the more they see the potential to serve not just Parkinson’s patients but provide support for a wide variety of medical ailments, like stroke patients or people rehabilitating from surgery.
“I love thinking about that. Yes, this is great for Parkinson’s – but it can also be applied to so many different conditions for rehabilitation,” Kandel said.
These are the principal investigators of the Parkinson’s Project:
- Gedas Bertasius, assistant professor of computer science.
- Nina Browner, Bryson Distinguished Professor of Neurology, Movement Disorders.
- Henry Fuchs, Federico Gil Distinguished Professor of Computer Science.
- Michael Lewek, associate professor, division of physical therapy.
- Danielle Szafir, assistant professor of computer science.
- Daniel Szafir, associate professor of computer science.
