Cool Jobs: Doing real science in virtual worlds | Science News for Students

Cool Jobs: Doing real science in virtual worlds

Meet scientists who use virtual reality to study language, cure phobias and move rivers
Apr 13, 2017 — 7:10 am EST
virtual reality

The virtual world can be easy to access with a little technology. And it’s more than just fun and games. Some scientists are using the virtual world for their research.


This is one in a series on careers in science, technology, engineering and mathematics made possible with generous support from Arconic Foundation.

Strap on a virtual reality headset and you’ll enter a different world. Without leaving your house, you can fly a spaceship through a make-believe galaxy. You can play pool with friends. Or you can perform surgery on an alien.

Virtual reality, or VR, uses special technology to trick the brain into thinking these experiences are real. A technique called stereoscopy (STAIR-ee-OSS-kuh-pee) sends a slightly different image to each eye. This can create the illusion of depth. It certainly makes video games feel more realistic. But VR isn’t just for fun. It also can help scientists do their research or share it with others.

Scientists are using VR to learn more about people and the planet. One engineer uses this technology to let kids build mountains and carve out rivers with their bare hands. A scientist who studies language puts people in a virtual restaurant to learn what happens in their brains as they converse. A doctor takes patients on a virtual field trip to swim with dolphins. The worlds they visit are not real, but the science is.

Decoding dialogue

David Peeters loved learning foreign languages when he was growing up. His first language was Dutch. He studied three others at school — German, French and English.

In college and graduate school he focused on linguistics. It’s the science of human language. And the more he learned, the more Peeters began to wonder what happens inside our brains as people converse. He began to look at language through the lens of neuroscience — the study of the brain.

“There’s a lot about the way the brain processes speech that we don’t understand,” he says. Peeters is a scientist at the Max Planck Institute for Psycholinguistics in Nijmegen (Nih-MAY-jin), a city in the Netherlands. Peeters studies the way people communicate. To answer some of his questions, he built a virtual restaurant.

Real people stroll into it. To do this, they wear 3-D glasses. The small room they walk in has screens on every wall. It’s called a cave automatic virtual environment — CAVE for short. It’s basically a theater with a 3-D movie projected on every wall. To someone wearing 3-D glasses, it feels almost like a real world. (For people familiar with Star Trek: Next Generation, CAVEs are essentially a real-life version of the holodeck.)

Story continues below image.

Virtual restaurant
This restaurant isn’t real, but it’s helping linguist David Peeters study language. He observes what happens in the brain as real people talk to virtual diners in this digital eatery.
Stefanie Kranaan

The screens show scenes inside the virtual restaurant. Each person who takes part in the study “becomes” a waiter or waitress through an avatar. That avatar is a make-believe character. It can be moved around and used to talk to others in a virtual world. Participants move their avatar simply by walking around the CAVE.

Peeters wants to find out what happens in people’s brains as they speak with virtual restaurant customers through their avatar. He does this by having each person wear a cap covered in electrodes.

These small sensors on wires attach to the outside of the head. Cells in the brain communicate with each other by sending tiny zaps of electricity back and forth. Electrodes listen for these electrical impulses and then report them to a computer. The computer records this brain activity as a set of wavy lines called an EEG. That’s short for electroencephalogram (Ee-LEK-troh-en-SEFF-uh-low-gram).

Peeters uses the EEG data to see which parts of the brain are most active during a conversation. This gives him clues about how the brain processes or understands different patterns of speech.

For example, there are direct and indirect ways to say something. “Please bring me another soup” is a very direct way to communicate a need, Peeters points out. But a lot of our conversations are indirect. In the virtual restaurant, a customer may simply say, “My soup is cold.”

“We understand this means the customer wants another soup, even though they didn’t ask for it,” says Peeters. That’s indirect language.

Peeters studies the differences in brain activity when a person hears direct versus indirect speech patterns. He hopes such research will one day help scientists better understand disorders such as autism. That’s a condition in which people have a hard time processing speech and communicating.

A new way to relax

For many years, Wim Veling used VR to help patients overcome phobias, or fears. As a psychiatrist, he treats patients with mental-health disorders. Veling works at the University of Groningen in the Netherlands.

A person with a fear of heights, for instance, might wear a VR headset and practice standing on top of a virtual building. When that person feels comfortable with a low-rise building, he or she can move to a higher one, Veling explains. This technique is called exposure therapy. It involves exposing people to frightening situations without putting them in true danger.  

Veling also treats patients with other mental-health disorders. These include depression and anxiety. Such patients can have problems with nervousness, irritability, sleeping and concentrating. “It can be very difficult to relax if you are feeling depressed or anxious,” he notes.

Some people with these disorders try therapy with animals to help them feel calm. One form, known as dolphin therapy, has patients swim with the marine mammals. But there can be drawbacks to this. Dolphins are big, strong animals. So swimming with them can be dangerous. Some people, of course, cannot swim . They may even be afraid of immersing themselves in open waters. This activity also can cost a lot of money if it requires traveling to where the dolphins live.

Veling wondered whether VR might offer a safer and easier alternative.

VR scientist
Psychiatrist Wim Veling talks to an avatar in virtual reality.
W. Veling

Simply watching a television show about dolphins can feel relaxing too. But VR might create a much more powerful response in the brain, says Veling.

When you put on a VR headset, a virtual world surrounds you on all sides: Schools of tiny fish pass by. Waves splash overhead. These images and sounds trick the brain into thinking you’re really in the water with dolphins swimming all around you, even though you’re sitting on a comfy couch. And, Veling adds, “When you’re really immersed in that environment, you begin to forget your worries.”

Veling teamed up with a filmmaker who uses a special VR camera. This camera can rotate 360 degrees to make virtual-reality movies. The filmmaker took the camera into the ocean to make a VR video of a pod of wild dolphins.

But Veling needed evidence that the video actually helped people. So he started a small experiment to see if virtually swimming with dolphins helped his patients relax. To test this, he had his patients watch the video through a VR headset. At the same time, Veling watches them for signs of stress. He counts how fast his patients’ hearts beat. He also measures whether they’re tensing their muscles and how much they sweat.

A rapid heartbeat, muscle tension and sweating are signs that someone is not very relaxed, he says. He cannot say with certainty that the VR sessions are helpful. He’s still gathering data. But he does hope to have some answers soon.

Veling plans to make the dolphin immersion video available to the public in the next year so that anyone with a VR headset can try it at home. He hopes it will be a new way for people to relax.

Story continues below video. 

Psychiatrist Wim Veling tests whether swimming with dolphins in virtual reality can help calm patients with depression and anxiety.
the Dolphin Swim Club

Moving rivers

In California, Jeanette Newmiller wants to solve a very different kind of problem. She’s a water-resources engineer at the University of California, Davis. Her mission? “Figure out the best way to get water where we need it, when we need it,” she says.

Sometimes, though, engineers like Newmiller get concerned with the fish in those waters. Newmiller is working with biologists on a project to move young salmon from the Sacramento River onto flooded rice fields. That may seem like an odd place for young fish, but it could help them reach adulthood.

The Sacramento River is California’s biggest. It runs from mountains in the North of the state through its Central Valley. From there, it flows to the Pacific Ocean through San Francisco Bay. Every spring, salmon are born in the Sacramento River. These small fry travel down the river and into the ocean where they will grow into adults. When it’s time to lay eggs, those adults will return to spawn in the river of their birth.

Water model
Jeanette Newmiller uses computer models to make predictions about what will happen to the flow of water in the Sacramento River if certain engineering structures are added. In this model, the yellow line represents the river. The green parts of the map show lower-elevation areas. The red areas that run alongside the river show areas of higher elevation or walls that keep the river from flooding.
Jeanette Newmiller

But several years ago, biologists noticed the baby salmon were reaching the ocean too quickly. They were too tiny when they got there. Small salmon are more likely to be eaten by predators such as orcas, sharks and seals.

The biologists showed, though, that they could fatten up the baby salmon by slowing their migration. They put young fish from a hatchery onto trucks. Then they drove these fish to flooded rice fields. The fields are purposely flooded in late winter to create a temporary home for the salmon. (That flooding doesn’t hurt the farm because it occurs during a part of the year when rice isn't grown.) In these fields, the baby salmon feasted on tiny bugs for a few weeks before being released into the Sacramento River. These young fish grew bigger than did salmon that swam straight out to sea.

The rice fields could be a pit stop for salmon on their way to the ocean. But it isn’t practical to take the fish in and out on trucks. So scientists need to reroute the water to carry the tiny fish there and away again. That’s where engineers such as Newmiller come in. She makes computer models to help design gates along the waterways that connect the Sacramento River to the rice fields.

How big should we make the gates? Where should we put them? How will water flow through them? Newmiller tests out different real-life scenarios with computer models.

She uses augmented reality (AR) to bring her computer models to life. Augmented reality is a technology that lets you see virtual images layered on top of real-life environments. (Think Pokémon GO.)

Newmiller combines AR technology with an ordinary sandbox. Another UC Davis researcher created the first AR sandbox. Newmiller made a version on wheels and takes it to classrooms. Kids can use their hands to push its sand into hills and valleys. A Microsoft Kinect 3D camera detects any changes to the sand. A projector displays colors on top of the landscape like a topographical map — one that depicts hills and valleys. If someone holds her hand over the sand like a raincloud, virtual water appears beneath it. If playful fingers carve out a riverbed or build a dam, the flow of the virtual water will follow a new path.

Newmiller uses the AR sandbox to teach kids and adults about her work. “People in California vote on water resources all the time. Teaching them about what I am doing helps them become better citizens,” she says.

This engineer hopes her AR sandbox might get kids interested in engineering too. Newmiller admits she had no idea what engineers did when she was a teenager. “As a result, I almost missed out on a really cool career. I want to show kids what we do,” she says. With her help, they won’t miss out.

Engineer Jeanette Newmiller uses an augmented reality sandbox to teach students about her work modeling the flow of rivers.
Center for Watershed Sciences

Power Words

(more about Power Words)

anxiety     A nervous reaction to events causing excessive uneasiness and apprehension. People with anxiety may even develop panic attacks.

augmented reality     A technology similar to virtual reality that places digital illusions into the real world.

autism     (also known as autism spectrum disorders ) A set of developmental disorders that interfere with how certain parts of the brain develop. Affected regions of the brain control how people behave, interact and communicate with others and the world around them. Autism disorders can range from very mild to very severe. And even a fairly mild form can limit an individual’s ability to interact socially or communicate effectively.

avatar     (in virtual reality) An electronic rendering of some image. It becomes the virtual form of some character (even a computer or game user) that will be moved and manipulated by a computer user. It can interact with its environment and other characters.

bug     The slang term for an insect. Sometimes it’s even used to refer to a germ.

CAVE    Short for cave automatic virtual environment. It's a blank-walled room into which a virtual environment is projected. People can walk through it, wearing virtual-reality head gear and other devices to seeming move, touch or experience the projected "world."

cell     The smallest structural and functional unit of an organism. Typically too small to see with the unaided eye, it consists of a watery fluid surrounded by a membrane or wall. Depending on their size, animals are made of anywhere from thousands to trillions of cells. Some organisms, such as yeasts, molds, bacteria and some algae, are composed of only one cell.

computer model     A program that runs on a computer that creates a model, or simulation, of a real-world feature, phenomenon or event.

depression     A low spot, such as in a field or the surface of a rock. (in medicine) A mental illness characterized by persistent sadness and apathy. Although these feelings can be triggered by events, such as the death of a loved one or the move to a new city, that isn’t typically considered an “illness” — unless the symptoms are prolonged and harm an individual’s ability to perform normal daily tasks (such as working, sleeping or interacting with others). People suffering from depression often feel they lack the energy needed to get anything done. They may have difficulty concentrating on things or showing an interest in normal events. Many times, these feelings seem to be triggered by nothing; they can appear out of nowhere.

disorder     (in medicine) A condition where the body does not work appropriately, leading to what might be viewed as an illness. This term can sometimes be used interchangeably with disease.

dolphins     A highly intelligent group of marine mammals that belong to the toothed-whale family. Members of this group include orcas (killer whales), pilot whales and bottlenose dolphins.

electricity     A flow of charge, usually from the movement of negatively charged particles, called electrons.

electrode     A device that conducts electricity and is used to make contact with non-metal part of an electrical circuit, or that contacts something through which an electrical signal moves. (in electronics) Part of a semiconductor device (such as a transistor) that either releases or collects electrons or holes, or that can control their movement.

engineer     A person who uses science to solve problems. As a verb, to engineer means to design a device, material or process that will solve some problem or unmet need.

environment     The sum of all of the things that exist around some organism or some device and the condition those things create for that organism or device. Environment may refer to the weather and ecosystem in which some animal lives, or, perhaps, the temperature, humidity and placement of components in some electronics system or product.

field     An area of study, as in: Her field of research was biology. Also a term to describe a real-world environment in which some research is conducted, such as at sea, in a forest, on a mountaintop or on a city street. It is the opposite of an artificial setting, such as a research laboratory.

fry    (in biology) Hatchling fish that are tiny and defenseless.

graduate school     A university program that offers advanced degrees, such as a Master’s or PhD degree. It’s called graduate school because it is started only after someone has already graduated from college (usually with a four-year degree).

hover     To above the ground, maintaining an intentional position.

illusion     A thing that is or is likely to be wrongly perceived or interpreted by the senses.

marine mammals     Any of many types of mammals that spend most of its life in the ocean environment. These include whales and dolphins, walruses and sea lions, seals and sea otters, manatees and dugongs — even polar bears.

migration     (v: to migrate) Movement from one region or habitat to another, especially regularly and according to the seasons. One that makes this move is known as a migrant.

model     A simulation of a real-world event (usually using a computer) that has been developed to predict one or more likely outcomes.

muscle     A type of tissue used to produce movement by contracting its cells, known as muscle fibers. Muscle is rich in protein, which is why predatory species seek prey containing lots of this tissue.

orca     A large species of black-and-white porpoise. Also known as the killer whale.

Pacific     The largest of the world’s five oceans. It separates Asia and Australia to the west from North and South America to the east.

phobia     An extreme or irrational fear of or aversion to something.

planet     A celestial object that orbits a star, is big enough for gravity to have squashed it into a roundish ball and has cleared other objects out of the way in its orbital neighborhood. To accomplish the third feat, the object must be big enough to have pulled neighboring objects into the planet itself or to have slung them around the planet and off into outer space. Astronomers of the International Astronomical Union (IAU) created this three-part scientific definition of a planet in August 2006 to determine Pluto’s status. Based on that definition, IAU ruled that Pluto did not qualify. The solar system now includes eight planets: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus and Neptune.

pod     (in zoology) The name given to a group of toothed whales that travel together, most of them throughout their life, as a group.

predator     (adjective: predatory) A creature that preys on other animals for most or all of its food.

psychiatrist    A medical doctor who spends many years learning to study and treat diseases of the human mind. Treatments may consist of talking therapies, prescription drugs or both. This medical field is known as psychiatry.

scenario     An imagined situation of how events or conditions might play out.

sea     An ocean (or region that is part of an ocean). Unlike lakes and streams, seawater — or ocean water — is salty.

sensor     A device that picks up information on physical or chemical conditions — such as temperature, barometric pressure, salinity, humidity, pH, light intensity or radiation — and stores or broadcasts that information. Scientists and engineers often rely on sensors to inform them of conditions that may change over time or that exist far from where a researcher can measure them directly. (in biology) The structure that an organism uses to sense attributes of its environment, such as heat, winds, chemicals, moisture, trauma or an attack by predators.

sharks     A family of primitive fishes that rely on skeletons formed of cartilage, not bone. Like skates and rays, they belong to a group known as elasmobranchs. Then tend to grow and mature slowly and have few young. Some lay eggs, others give birth to live young.

stress     (in psychology) A mental, physical, emotional, or behavioral reaction to an event or circumstance, or stressor, that disturbs a person or animal’s usual state of being or places increased demands on a person or animal; psychological stress can be either positive or negative.

technology     The application of scientific knowledge for practical purposes, especially in industry — or the devices, processes and systems that result from those efforts.

therapy     (adj. therapeutic) Treatment intended to relieve or heal a disorder.

virtual     Being almost like something. An object or concept that is virtually real would be almost true or real — but not quite. The term often is used to refer to something that has been modeled — by or accomplished by — a computer using numbers, not by using real-world parts. So a virtual motor would be one that could be seen on a computer screen and tested by computer programming (but it wouldn’t be a three-dimensional device made from metal).

virtual reality     A three-dimensional simulation of the real world that seems very realistic and allows people to interact with it. To do so, people usually wear a special helmet or glasses with sensors.

wave     A disturbance or variation that travels through space and matter in a regular, oscillating fashion.


  • MS-LS1-8
  • MS-LS2-2
  • MS-LS2-4
  • MS-LS2-5
  • HS-LS2-7
  • HS-LS2-8


Meeting: B. Brada. The Dolphin Swim Club – a healing virtual reality experience. Talk presented at VRPain 2016. September 19, 2016. Los Angeles, Calif.

Meeting: D. Peeters. Virtual reality revolution in the language sciences. Talk presented at Tilburg Center for Cognition and Communication. September 7, 2016. Tilburg, the Netherlands.

Report : J Katz et al. The experimental agricultural floodplain habitat investigation at Knaggs Ranch on Yolo Bypass 2012-2013. Preliminary report to the U.S. Bureau of Reclamation. October 1, 2013.

Further Reading

Questions for ‘Cool Jobs: Doing real science in virtual worlds’

Wordfind (click here)