Caroline Stefani uses the Confocal VR program in the lab. In the background, an image of what she is seeing through the headset. (GeekWire Photo / Clare McGrane)
Caroline Stefani has a cool job: She works at the Benaroya Research Institute at Virginia Mason in Seattle, looking for new ways to treat diseases like multiple sclerosis and Type 1 diabetes. To do that, Stefani spends a lot of time taking pictures of tiny, tiny cells through a microscope, then trying to imagine those images in three dimensions.
It’s not easy. In fact, it turns out a lot of scientists have a hard time wrapping their minds around the tiny, three dimensional structures that they work with every day, like cells and proteins.
Luckily, Stefani’s office is right next door to Tom Skillman, who leads Benaroya’s research technology efforts. Talking with Stefani over drinks one evening, Skillman had an idea: using data from microscopes, he could build a full 3D model of the cells she studies — in virtual reality.
A year later, the lab that Stefani works in has a fully-operational VR program that lets her and other researchers see their work like never before. It’s just one of many ways that virtual and augmented reality are making waves in the world of medicine and medical research.
Adam Lacy-Hulbert, who runs the lab Stefani works in, says the program gets to the heart of a scientist’s work. Although it can seem flashy, medical research mostly means trying the same thing over and over — and over — again.
,
,
Medical researchers Caroline Stefani (left) and Adam Lacy-Hulbert (center) have been using a VR system to see immune cells in 3D. It was developed by research technology lead Tom Skillman (right). (GeekWire Photo / Clare McGrane)
“A lot of it is looking, noticing, and trying again,” Lacy-Hulbert said. His lab studies autoimmune diseases, where a person’s immune system attacks parts of their body. He said the scientists spend a lot of time trying to understand exactly what’s going on in a single cell or an interaction between cells.
“That’s what’s been so cool about it — it’s almost immediate,” he said about the VR program.
Skillman, who comes from a technology background, said it has been incredible to see the scientists at Benaroya use the program.
“I wish we could capture one of the times when one of these scientists look at it for the first time,” he said. “They’re just giddy with excitement because they’re seeing things they never saw before.”
Here’s how the program, called Confocal VR, works: It starts with images taken in a process called confocal microscopy. Those images look like the one below, a 2-dimensional slice of the cell taken on different planes. It’s similar to an MRI that you would get in a hospital.
These images are of a dying cell, in green, being eaten by an immune cell. The immune cell’s nucleus is blue and its digestive tract is red.
,
,
A confocal microscopy image of a dying cell (in green) being eaten by an immune cell. The immune cell’s nucleus is colored blue, its digestive tract is red. (Benaroya Research Institute Image)
Although the images help us understand roughly where the cells are and how they are interacting, it’s hard to get a sense of depth or see precisely what’s going on. Even a 3D compilation of the images is challenging to understand through a computer monitor.
Those images are then transferred to Confocal VR, which Skillman built using Unity, the popular video game making software. The process is fast: Lacy-Hulbert said it basically works in the amount of time it takes for him to walk from the microscope to the VR headset.
“You have a quick look at your sample in the microscope, and then almost the first time you actually interact with it is in VR,” he said.
Skillman and Stefani worked together on the program, alongside another researcher in the lab, Mridu Acharya. They used what the tech world calls rapid prototyping.
“I would build something, give it to Caroline. She would use it, give me feedback. I would go off, adjust it, and we just iterated as fast as we could,” Skillman said.
The end product is rather stunning: Not only does the program show scientists cells in new ways, it’s also incredibly intuitive to use.
Personally, I’ve never used a confocal microscope before, nor have I studied the immune system in any depth. When Stefani explained the details of the 2D image, I had trouble following her. But as soon as I put on the lab’s Vive headset, I instantly understood the details she had described. I started noticing more things and asking the researchers questions about different parts of the cells.
You can see a demo of what I saw in the video below:
,
Source: Geek Wire