What Is Augmented Reality?

The word "augment" means to increase, extend, or make better. Augmented reality (AR) can be understood as a type of extended reality (XR) where the real world is expanded or enhanced through the use of virtual elements, usually overlaying those elements on the view of the real world through the use of a visual device.

AR can work in several different ways and is used for many different reasons, but in most cases augmented reality virtual objects are overlaid on and tracked in a view of the real world, creating the illusion that they occupy the same space.

Augmented reality devices have a display, input device, sensor, and processor. These devices can be monitors, head-mounted displays, eyeglasses, contact lenses, gaming consoles, and even smartphones, among others. Sound and touch feedback can be included in an AR system, as well as through other non-visual methods and devices.

Although augmented reality is somewhat similar to other forms of XR, like virtual reality (VR), it's distinctly different. Virtual reality is an experience that's entirely simulated—both the view of "reality" and the objects in it—while AR only uses some virtual aspects, which are mixed into reality to form something different.

How Augmented Reality Works

Augmented reality is live. For it to work, the user must be able to see the real world as it is right now. AR manipulates the real world space the user sees, altering their perception of reality.

In one form of AR, the user watches a live recording of the real world with virtual elements imposed on top of it. Lots of sporting events utilize this type of AR; the viewer can watch the game live from their own TV, but also see the scores overlaid on the game field.

Another type of AR allows the user to look around their environment normally and in real time, but through a display that overlays information to create the augmented experience. An example might be a headset that appears much like a regular pair of glasses, but through the screen, the user can see GPS directions on the road or sidewalk as they drive or walk, or see the current temperature in the sky when they look up.

When a virtual object is placed between the user and the real world, object recognition and computer vision can be used to allow the object to be manipulated by actual physical objects, and allow the user to interact with the virtual elements.

For example, some retailer mobile apps let shoppers select a virtual version of something they're considering purchasing, such as a piece of furniture, and view it in the real space of their home through their phone. They can see their actual living room, for example, but the virtual couch they’ve chosen is now visible to them through their screen, letting them decide if it will fit in that room and if they like the look of it within that room.

Another example allows customers to scan products or special codes (such as UPC symbols) that use augmented reality to show the customer more information about a physical product before they buy it, see reviews from other buyers, or check what’s inside their unopened package.

Marker and Markerless AR

When object recognition is used with augmented reality, the system recognizes what's being seen and then uses that information to engage the AR device. It's only when a specific marker is visible to the device that the user can interact with it to complete the AR experience.

These markers might be QR codes, serial numbers, or any other object that can be isolated from its environment for the camera to see. Once registered, the augmented reality device might overlay information from that marker directly on the screen or open a link, play a sound, etc.

Markerless augmented reality allows a system to use a location or position-based anchor points, like the compass, GPS, or accelerometer. These types of AR systems are implemented when the location is key, such as with navigation.

Layered AR

This type of AR uses a device to recognize a physical space and then overlay virtual information on top of it. It's how you can try on virtual clothes, display navigation steps in front of you, check whether a new piece of furniture will fit in your house, put on fun tattoos, and more.

Projection AR

This might seem at first to be the same as layered or superimposed augmented reality, but it's different in one specific way: actual light is projected onto a surface to simulate a physical object. Another way to think of projection AR is a hologram.

One specific use for this kind of augmented reality might be to project a keypad or keyboard directly onto a surface, allowing a user to type using the virtual keyboard.

There are many advantages to using augmented reality in areas like medicine, tourism, the workplace, maintenance, advertising, the military, and others.

AR in Education and The Workplace

In some senses, it can be easier and more fun to learn with augmented reality, and there are tons of AR apps that can facilitate that. A pair of glasses or a smartphone can be all you need to learn more about physical objects around you, like paintings or books.

One example of a free augmented reality app is SkyView, which lets you point your phone at the sky or the ground and see where stars, satellites, planets, and constellations are located at that exact moment, both during the day, at night, and from the opposite side of the planet.

SkyView is considered a layered augmented reality app that uses GPS. It shows you the real world around you, like trees and other people, but also uses your location and the current time to teach you where these objects are located and give you more information about each of them.

Google Translate is another example of an AR app useful for learning. With it, you can scan text in a language you don’t understand, and it will translate it for you in real time. When viewed through the app on a phone, it looks like the physical space that occupies that text has changed to the target language.

Job education, too, is changing due to AR. Doug Stephen, president of CGS Enterprise Learning division, says it's becoming part of on-the-job training options.

"Often considered an emerging and disruptive technology, [AR] provides learners with an immersive format," he says. "An example of how to use consumer-focused AR for education is use of mesh modems that homeowners install to extend their internet reach. By using AR, the person can visualize the internet strengths throughout the home on a tablet or mobile device. This can alleviate in-home service (because) customer support is better enabled to see what the customer sees.

"It can also offer immediate education on set up and installation as well as help the user determine more-effective modem placements to optimize the range of the internet connections. This will ultimately help the consumer save time, effort, money and potential frustration from waiting for a technician to do in-home service."

AR in Navigation

Displaying navigation routes against a windshield or through a headset delivers augmented instructions for drivers, bicyclists, and other travelers so that they don’t have to look down at their GPS device or smartphone just to see which road to take ahead. Pilots might use an AR system to display transparent speed and altitude markers directly within their line of sight for much the same reason.

Another use for an augmented reality navigation app might be to overlay a restaurant’s ratings, customer comments, or menu items right on top of the building before you go inside. It might also show you the quickest route to the nearest Italian restaurant as you walk through an unfamiliar city.

GPS AR apps like Find Your Car with AR can be used to find your parked car, or a holographic GPS system like WayRay might overlay directions right on the road in front of you.

AR in Games

There are lots of augmented reality games and toys that can merge the physical and virtual world, and they come in many different forms for lots of devices. One well-known example is Snapchat, which lets users overlay fun masks and designs on their faces before sending a message. The app uses a live version of your face to put a virtual image on top of it.

Other examples of augmented reality games include Pokemon GO!, INKHUNTER, Sharks in the Park, Sketchar, Temple Treasure Hunt Game, and Quiver.

AR vs MR

Mixed reality (MR), as the name suggests, mixes real and virtual environments to form a hybrid reality. MR uses elements of both virtual reality and augmented reality to create something new. It's tough to categorize MR as anything but AR because it works similarly, by overlaying virtual elements directly onto the real world and letting you see both at the same time.

However, one primary focus with mixed reality is that the objects are anchored to real, physical objects that may be interacted with in real time. This means that MR could allow virtual characters to sit in the actual chairs in a room, or for the virtual rain to fall and hit the actual ground with life-like physics.

Mixed reality lets the user seamlessly exist in both a real state with the real objects around them and the virtual world with software-rendered objects interacting with real-world objects to create a fully immersive experience. One good example of what's meant by mixed reality is Microsoft HoloLens.