What is VR Passthrough and How Does it Work?
Virtual reality headsets have come a long way since products like the Oculus Quest burst into the scene a few years ago. Increasingly, both consumers and enterprise users are searching for more versatile immersive solutions, and features like VR passthrough are becoming commonplace.
In the past, whenever you wore a virtual reality headset, you’d basically block out the real world around you, diving completely into a new digital landscape. Now, virtually all top headsets feature some passthrough feature, allowing you to connect the virtual and real worlds.
At this point, full “VR” headsets are becoming increasingly less common, and mixed reality is soaring into the spotlight. But what exactly is VR passthrough, how does it work, and what does it mean for the future of extended reality? Here’s everything you need to know.
What is VR Passthrough?
VR passthrough is a popular feature within virtual reality headsets, which allows users to see “through” their virtual displays and connect with the real world around them while still maintaining access to digital content and experiences.
It’s a feature that VR headset vendors have been experimenting with for some time, for multiple reasons. Initially, early headsets like the Meta Quest 2 only offered basic passthrough features, allowing users to see their surroundings displayed in front of them in black and white.
Now, innovators are leveraging more advanced sensors, cameras, and intelligent systems to bridge the gaps between virtual and physical worlds like never before. The VR passthrough features on the Apple Vision Pro, for instance, are strong enough to eliminate the need to remove your headset entirely when you want to reconnect with the real world.
VR passthrough isn’t just a great way to improve the “safety” of virtual reality experiences, ensuring you can see what’s going on around you in an instant. This functionality is also allowing vendors to create entirely new immersive experiences for users. For instance, with VR passthrough, you can sit at your desk in the office, interact with team members, and see digital content at the same time, taking productivity and creativity to the next level.
How Does VR Passthrough Work?
As mixed reality experiences become increasingly popular in the XR world, vendors are experimenting with new technologies to enhance VR passthrough. These days, cutting-edge systems take advantage of built-in AI software and spatial mapping capabilities, to deliver a more precise and accurate intersection between digital content and real-world environments.
On a basic level, however, VR passthrough works by using cameras and sensors mounted onto a headset to capture your physical surroundings. When you look at the world around you in “passthrough” mode, you’re not actually looking “through” the headset.
Instead, you’re looking at camera footage streamed on the display in front of your eyes in real time. When the cameras on your headset capture the world around you, they use clever software to stitch feeds together, patch up blind spots, and correct perspectives, all within an instant.
The software that enhances the visual feed you see helps to minimize motion sickness and distortion, making the transition between virtual reality and the real world more comfortable.
Aside from allowing you to interact with your physical surroundings (so you can navigate furniture without bumping into it), VR passthrough also overlays digital objects onto your surroundings. For instance, you might be able to see an avatar sitting beside you in a meeting room or a three-dimensional prototype of a product placed on a table.
What are the Benefits of VR Passthrough?
Initially, when companies started experimenting with more simplistic VR passthrough experiences (like black and white passthrough), most people saw the feature as a novel “add-on” to VR. Enabling passthrough modes meant that vendors could give users a safer experience within VR, enhance the usability of headsets, and boost their practicality for everyday use.
Today, VR passthrough is becoming increasingly essential for a few reasons. First, while many consumers and enterprise users love immersing themselves in full VR experiences, particularly for training and entertainment, we don’t always want to cut ourselves off from the real world.
That’s particularly true in the enterprise landscape. When team members are experimenting with VR, perhaps to build prototypes of products or collaborate on projects, they can’t disconnect from the real world entirely. They still need to be able to use office resources, interact with the team members around them, and so on. VR passthrough enables this functionality.
With state-of-the-art passthrough experiences, users benefit from:
Enhanced Creativity
While team members can certainly dive fully into a virtual reality experience to build products and prototypes, using passthrough during creative projects has many benefits. It can allow users to get a more realistic insight into the size and scale of the products they’re developing and maintain productivity. For instance, imagine you were working on creating a new computer system, but only a fraction of the materials you needed were actually available in the room.
With VR passthrough, users can combine their real-life surroundings with virtual 3D images, experimenting with digitized materials and design elements in real-time. This makes it easier to visualize problems, and come up with creative solutions at speed.
Seamless Interaction with Reality
Full immersion in virtual reality is fun, and ideal for certain, specific use cases. However, in the enterprise world in particular, it’s usually impractical for users to be cut off from their physical surroundings for long periods. For instance, if you were working in VR on a project and didn’t have access to VR passthrough, you’d have to constantly remove your headset.
Every time you needed to speak to a colleague, check the details on a document, read an email, or answer a text, you’d have to take a break from VR. Even grabbing a cup of coffee or a glass of water would be difficult without passthrough. The ability to seamlessly reconnect with the real world can save users a lot of time and effort, making VR more practical.
Enhanced Safety
As mentioned above, one of the biggest benefits of VR passthrough is that it makes VR safer for everyday users. It’s easy to lose track of where you actually are in a physical space when you’re in VR. Even software solutions, like the Meta Quest Guardian, aren’t perfect at detecting the furniture, objects, and moving people around you at all times.
Passthrough ensures you can stay safe, and aware of what’s going on around you, even when you’re accessing virtual reality apps. This is particularly important now that people aren’t just using their VR headsets at home, or in dedicated office spaces. Increasingly, people are wearing headsets on the move, taking them outside, or into the field in the manufacturing space.
Without VR passthrough, navigating the real world with a headset on would be extremely dangerous. Although passthrough doesn’t eliminate the dangers of wearing your headset outside entirely, it does reduce the risk facing everyday users.
The Issues with VR Passthrough Today
While VR passthrough technologies are becoming more commonplace, powerful, and intuitive, they’re still far from perfect. We’ve definitely come a long way from simple black-and-white passthrough options with grainy, inaccurate images.
These days, leading vendors like HTC VIVE, Varjo, and even Meta are leveraging more powerful cameras, sensors, and software to align the real and digital worlds with excellent precision. However, the quality of a “passthrough” experience can still be extremely variable from one headset to the next.
Some, more affordable mixed reality headsets, still rely on reasonably low-quality cameras, similar to those you’d find on a smartphone. This means you won’t always get a “true to life” image of the world around you. A lot of devices still struggle with issues with depth of field and aperture which can also make it difficult to accurately determine where items are in your surroundings.
You might reach out to touch a table and find that it’s actually a few centimetres further away than you originally thought. Higher-quality sensors and software would address this issue, but they also mean vendors have to invest in higher-quality processors, stronger batteries, and more advanced tech. All of this would lead to more expensive, heavier, and potentially less practical headsets.
The Future of VR Passthrough Technology
The good news for consumers and enterprise users ready to dive into the new era of “mixed reality”, is that VR passthrough technologies are improving. Companies like Meta and Apple are working on implementing AI technologies and spatial computing systems into their headsets which allow for a more precise and accurate alignment between digital and real-world spaces.
At the same time, the quality of the screens and cameras in headsets is evolving, allowing for crisper, more true-to-life images. Companies like HTC VIVE are even using IR technologies and light depth sensors to boost positional tracking capabilities.
We’re also seeing an increase in companies experimenting with ways to make “interacting” with digital objects more immersive in mixed reality. Hand, finger, and eye tracking technologies are becoming more advanced and sophisticated.
Some companies are even experimenting with haptic systems, like gloves and body suits which can track minute user movements, even without relying entirely on camera feeds. For instance, the haptic wristband included with the Meta Orion prototype glasses are a great example of the types of EMG-powered solutions that could transform he future of mixed reality.
VR Passthrough: Creating the Future of Mixed Reality
Ultimately, the continued growth of the VR landscape and the future of the metaverse will depend on how well hardware manufacturers can adapt to a growing demand for practical, immersive, and natural user experiences. VR passthrough technologies and advanced spatial computing systems will become increasingly essential in headsets in the years ahead.
Already, passthrough technology has evolved from being a “nice to have” feature in cutting-edge headsets, to a standard part of almost every VR system. Even budget-conscious solutions, like the Meta Quest 3S, now have VR passthrough capabilities as standard.
As we move forward into the future of extended reality, static virtual reality experiences will likely become a thing of the past, and mixed reality will become the new normal.
Quelle:
