Cleveland Clinic first to use augmented reality in this way
We loved seeing the ghostly image of Princess Leia plead for help in the first Star Wars flick.
Cleveland Clinic is a non-profit academic medical center. Advertising on our site helps support our mission. We do not endorse non-Cleveland Clinic products or services. Policy
Today, the same fascinating hologram technology has found its way into medicine. Cleveland Clinic is first to use augmented reality, or AR, in face transplantation.
Rather than create an artificial world like virtual reality, AR enhances the environment you see — with a computer’s help.
In the first total face transplant here in 2017, the computer was lodged in a headset surgeons wore.
It superimposed holographic images of hidden structures onto what remained of a gunshot survivor’s face — along with images of the donor’s anatomy — to guide the transplant team.
Their goal: to give the young woman, Katie Stubblefield, the opportunity to eat, swallow, speak and show emotion once again.
“It was like having X-ray vision,” says Frank Papay, MD, Chairman of the Dermatology & Plastic Surgery Institute and a leader of the surgical team.
“You put the AR headset on and saw the patient’s anatomy in 3-D. You could walk around the holographic image, and interact with it. As a surgeon, I’d never had this amazing perspective.”
It took more than a year of collaboration for the medical team and biomedical engineers, led by Karl West, MS, to program the unique applications for the Microsoft HoloLens®.
In any transplant, surgeons must join the donor’s blood vessels and nerves to the recipient’s so the graft will “take.”
In face transplant, they must join intricate facial vessels and nerves, and — since everyone’s facial bones are unique — carefully align skull, jaw and other bony structures.
To make this happen, AR helped the surgeons visualize structures they couldn’t see, and decide how much tissue and bone to remove from Katie and her donor.
Before the transplant, the biomedical engineers combed through data from computed tomography (CT) scans of Katie and the donor.
“We imported only critical information from the CT scans into the AR headset to create the 3D reconstructions — it’s surprising how much information CTs contain,” says West. “Then a special lens projected those holograms onto the surgeons’ goggles.”
Once a donor was found, West’s team had about 24 hours to generate similar 3D holograms of the donor’s anatomy.
“We worked all night to extract the most important information from the donor’s CTs,” he says.
Dr. Papay adds, “It really helped us see how the skull, bony structures and blood vessels fit together in different scenarios. Seeing blood vessels deep in the skull was especially critical.”
The transplant team overlaid the holographic images onto Katie’s face to adjust their approach throughout the surgery.
To help with visualizing the procedure, the transplant team created exact plastic replicas of the recipient’s and donor’s heads with a 3D printer, a process that takes many hours.
“We’ve used 3D-printed models, which are actual size, in all our face transplants. They’re good for visualizing larger structures,” says Dr. Papay.
“AR is much better at visualizing small structures, like blood vessels, and displaying different scenarios to guide our surgical strategy.”
Dr. Papay sees Cleveland Clinic’s pioneering use of AR in face transplantation as an innovation that has found its place.
“Face transplant is a natural application of this technology, and we’re just getting started,” he says.
AR is no longer futuristic technology “from a galaxy far, far away.” It’s advancing medicine right here, right now.
Learn more about our editorial process.
Gunshot wound survivor's recovery continues
More than 20 organs and tissues can be donated, and one donor can help more than 80 people
Rest, hydration, protein and light exercise all play an important role in your healing
Having overweight and disordered eating is a high-risk combination that often gets dismissed or overlooked
Puffing on cigarettes is the leading cause of bladder cancer
Go ahead and get goopy to help boost hydration and repair damaged skin
Preserving your social life and protecting your mental health are key to living well with vision loss
Treatments are designed to help with growth hormone deficiency, which can affect your child’s height
Your metabolism may torch 1,300 to 2,000 calories daily with no activity
A gentle touch in all the right places may help drain your sinuses
