“Basically this all started with hyper-nerdism,” said Ben Erwin, describing his recent project—creating virtual reality (VR) simulations of deep-sea features. Erwin is one of MBARI’s ROV pilots, which means he “flies” remotely operated vehicles as they explore, map, and collect samples in the deep sea. He operates machinery in an environment that most people can’t imagine, much less visit. But he wants to take his surreal surroundings one step further, into the realm of virtual reality.

“VR has always captivated me, and when decent VR headsets were finally released in 2016, the time seemed right,” Erwin said. Virtual reality is an emerging technology that allows users wearing a headset and other equipment to experience and manipulate their computer-generated surroundings. Citing sci-fi movies and television shows as his inspiration, Erwin quickly adopted VR as an exciting addition to the astounding technology that surrounded him.

Erwin works on MBARI’s research vessel Western Flyer, a custom-built ship that serves as a host vessel for the ROV Doc Ricketts. During an ROV dive, Erwin and his colleagues work in the ROV control room, a space filled with computers, banks of monitors, buttons, knobs, levers, and a celebratory disco light for big discoveries; it already looks like something you might find on the bridge of a spaceship.

But Erwin sees a lot of potential to incorporate new VR technology into different aspects of deep-sea research.

In 2017, Erwin worked with Lonny Lundsten, an ecologist and research technician at MBARI, to create three-dimensional (3D) views of underwater objects, using a program called PhotoScan to triangulate the contours of the objects from different angles.

To create a 3D image, pilots fly the ROV all the way around an object, collecting high-definition video of all sides. Erwin then takes screen grabs from the video and assembles those still images in PhotoScan. The program tracks individual pixels from one photo to the next, slowly gathering information about where each pixel might be in space. The technical name for the process is structure-from-motion photogrammetry, similar to the technology used by Google satellites to map buildings in 3D.

For researchers, this could be an incredibly useful tool. For example, biologists are often interested in the amount of biomass (living material) in a particular habitat. A 3D scan of a rocky outcrop allows special software to virtually “remove” the rock, so that scientists can measure biomass growing on top of it.  Similarly, scientists can use the software to examine and measure delicate deep-sea creatures, like jellies and squids without the damage or disturbance caused by physical collection.

Erwin explained, “I’ve created a 3D VR experience where you can teleport yourself inside a larvacean’s mucus filter.” Larvaceans are small tadpole-shaped animals that create delicate and expansive mucus “houses” which they use to collect marine snow. MBARI bioengineer Kakani Katija uses lasers to study water flow through larvacean houses without capturing and destroying them; Erwin created an interactive model from her scans. “I made it about the size of a building, so you can virtually walk through it,” he said. “Who knows what insights a scientist can gain when their perspective shifts from a computer screen to this kind of virtual experience?”

Erwin has greater ambitions for the new technology. In the next few years, he hopes to secure funding and equipment that might help pilots and scientists become immersed in the underwater environment of the Doc Ricketts. “It’s sort of an Iron Man concept,” he said, referring to the comic book superhero’s ability to manipulate his robotic suit using virtual screens and controls. For ROV pilots, the dream is customizable controls and better maneuverability. For both pilots and scientists, the main goal would be a seemingly simple feature: the ability to see 360 degrees around the ROV. With the current setup, a pilot has to turn the entire SUV-sized device around to catch a glimpse of a creature swimming by just out of the camera’s frame.

“My hope is to get a panoramic 3D camera system installed on the Ricketts,” Erwin said. As part of a future deep-sea exhibit at the Monterey Bay Aquarium, Erwin wants visitors to be able to virtually ride along with the ROV thousands of feet below the ocean’s surface, gaining a sense of scale and immediacy difficult to capture in video and photos. With virtual reality becoming more accessible, technology is catching up with science fiction.

Article by Vicky Stein

Here is one of Erwin’s 3D models. Find others at Sketchfab.

For additional information or images relating to this article, please contact: Kim Fulton-Bennett
831-775-1835, kfb@mbari.org