Technology entrepreneur Raven Zachary has seen things he has difficulty talking about.
“It is a profound moment to put on the device and see an object on your table, and then to walk around that object and see that it is fixed there and behaving as if it were real. It is impossible for me to communicate the experience. You have to experience it for yourself,” says Zachary, cofounder of software firm Object Theory.
Fellow developer Ryan Peterson, head of Finger Food Studios, is a little more grounded. “The Star Trek: Next Generation holodeck is real today,” he says. “You can sit down with any corporation and come up with 20 ideas for using it that would transform their business.”
They’re talking about the latest generation of head-mounted displays that let the wearer interact with an enhanced version of the real world.
What the headsets offer is not virtual reality (VR), where the user is presented with an entirely artificial visual experience by a head-mounted display that typically blocks out the rest of the world. Instead, these devices provide augmented reality (AR), which adds information to your environment—images are projected against the back of the goggles, but you can continue to see through the goggles to the real world beyond.
Object Theory and Finger Food Studios use Microsoft HoloLens, which is currently available in developer and enterprise versions. But there is already competition on its way, including the Meta 2 from Meta Co., and a stealth product from a company in Florida called Magic Leap.
AR evolved
Analyst J.P. Gownder at Forrester Research traces the new AR devices to the ill-fated Google Glass (released in beta in 2013, withdrawn in 2015), which he calls “monocular glance-able smart glasses,” displaying information that the user could read by glancing to one side.
But while Google Glass is now widely considered obsolete, at least one firm is still enthusiastic about the technology. Google Glass produces “incredible benefits in quality, productivity, and employee safety,” says Peggy Gulick, director of business process productivity at farm implement maker AGCO Corp. in Jackson, Minn. Employees once had to climb to the top of new tractors to run tests, and then climb down to log the results on a computer. With Google Glass, the computer goes with them, hands-free, she says.
For the most part, though, Glass and its ilk were followed by “immersive AR smart glasses” such as Epson’s Moverio, which can add data to a scene and so is useful for guided museum tours, Gownder notes.
And now there’s a third generation—beginning with the HoloLens, Meta 2 and Magic Leap—which Gownder calls “holographic AR smart glasses,” also known as mixed reality (MR) headsets. These devices not only project images onto the user’s field of view, they use robotic vision to detect features of the environment and generate images (i.e., holograms) to fit within those features. If, for instance, a holographic object or person is supposed to be standing behind a desk, the desk will correctly block the view of the hologram. If the user walks around the desk, the view of the hologram will change as the user moves.
Shared realities
This new generation of AR devices presents a mix of real and virtual images that interact realistically, and which the user can share with others. Achieving this result involves a mashup of vision, display and robotic technologies.
According to Meta vice president Ryan Pamplin, by using stereoscopic cameras, “Meta 2 maps the room around you using SLAM [simultaneous location and mapping].” SLAM produces a field of view that is so wide that headset wearers can’t see the edges, which helps them forget they’re wearing glasses, he adds.
“Through computer vision it creates a 3D map of the room, using a point cloud that can be made into a mesh and kept in memory.”
Users in different locations can share the virtual room and its holograms. Mutual presence takes a connection of about 10 megabits with the Meta 2, says Pamplin. “It’s like standing in front of each other. You can see the whole body of the other person. You can converse and push [holographic] objects back and forth. It’s not photo-realistic—you know it’s a hologram—but you are obviously talking to a human.
“For great 3D, you need a late-generation midrange graphics card, such as you can get on a laptop for under $1,000, or even less on a desktop,” Pamplin adds. “You can run it from a pocket machine if you don’t need fancy graphics.”
FingerFoot Studios
Users examining holographic airflow indicators projected onto a semi-truck with the Microsoft HoloLens system. The small truck in the foreground is entirely a hologram.
As for the Microsoft HoloLens, Object Theory’s Zachary estimates that, tethered to a PC with game-level graphics circuitry, it should be able to handle up two million polygons, allowing enhanced realism without latency problems. Left to itself, the HoloLens can only handle about 150,000 polygons. “But 3D artists are familiar with the process of decimation, where you simplify an image or turn it into a pipe frame, or otherwise make a low-polygon model look compelling,” he says.
“We are now working with clients who have large amounts of 3D content, such as architects or product designers, who want to visualize that content in 3D,” notes Zachary. “They want to have the information projected [onto a headset], rather than sharing a flat blueprint on a Skype session.” With remote collaboration tools, users can talk about the object while rotating it or even walking into it, and while each person remains in his or her office. “Your brain buys into the idea that you are in two places at the same time,” he says.
At AGCO they have started experimenting with using a HoloLens to show welders where to place welds by displaying diagrams, notes Gulick. “We would like to get to the point where we have crosshairs [to indicate weld placement], but this is far better than nothing. We are not going back. AR solves so many problems of job training and cross training that manufacturers struggle with,” she says.
Oliver Diaz, head of Fuel.tech, offers AR systems based on the HoloLens to help train employees in the energy industry. “The system can recognize the equipment around you, query the web for maintenance procedures, take you through them step by step, and verify that you did them correctly,” says Diaz. “Corporations spend millions writing procedures but those live in three-ring binders and few people follow them. With our system, it takes 15 to 20 minutes to develop a procedure to show how to do something like changing the ink in a printer—it’s as easy as writing it in word processing.”
Future holograms
While the latest generation of AR headsets has not reached the end-user market yet, pundits are already worried about future software compatibility. With three competing expressions of the same technology, adopters could end up with thousand-dollar devices that become non-standard.
In response, Zachary says the best bet is to use AR applications developed with the Unity game engine from Unity Technologies, as it supports a wide range of platforms. “But development is difficult at this stage because the industry is so new and there are not a lot of mature code libraries.” Zachary notes. “You must do everything from scratch. You must experiment and try interesting solutions, and some of them will fail,”
The next generation, meanwhile, is expected to include a feature called foveated rendering, explains Jim Preston, director of business development at Fove Inc., a Tokyo-based eye-tracking system startup. By tracking exactly where the eye is looking (as opposed to the direction the head is turned), the system can provide simulated depth of field, with whatever the user is looking at presented in maximum detail while the rest of the visual field is blurred. This matches the way the eyeball works, with maximum resolution around the focal point, called the fovea, Preston says.
Foveated rendering provides a more realistic visual experience, but also saves computational power, since much of the scene is rendered at lower resolution, he notes. Because eyeball tracking requires infrared emitters and detectors and an array of mirrors, it is easier to accomplish with an enclosed VR headset. He expects to see it on VR devices next year and on AR devices in a couple of years—unless Magic Leap (which won’t say) is already doing it.
Pamplin says foveated rendering is mostly for games, adding it would currently take too long for business app developers. “You need three focal planes for each eye, so you need to render each scene six times. Everyone will eventually have it, but meanwhile people want products today.”
In the end, it’s clear that Augmented Reality is just starting to become useful for businesses—but that it has a great deal of potential for the future. “Not everyone will be switching to it tomorrow,” says Gownder. “The difference today is that there are [industrial] scale players who have reached the point where they are showing their pilots. It’s not a fad. You can talk to firms using it in real environments today.”
This story, "Augmented reality: Next-gen headsets show business promise" was originally published by Computerworld.