IT/AV, Video

Augmenting Video Streams: Low-cost tools open new worlds

arctic home augmented reality

The enormous success of Pokémon GO has made millions of people aware of the power of augmented reality to engage and mobilize audiences. Although Pokémon GO was not the first treasure hunt AR (augmented reality) app (see the Greek4Food campaign for Yoplait by Total Immersion), it has accelerated AR interest in retail, training, tourism and gaming.

Hundreds of AR Applications

In addition to the use of AR in the training of astronauts, there are now hundreds of AR applications in aerospace, automotive, manufacturing, tourism, retail, digital signage and other fields. Major users include Coca Cola, Siemens, L’Oreal, Lowe’s, Hallmark, Paramount Pictures, Toyota and the American Museum of Natural History. Even those who have never used AR firsthand to select furniture or find a planet in the night sky have probably been exposed to the amazing potential of this technology through movies, such as Iron Man and Minority Report.

What exactly is augmented reality, and why is it of significance to video, AV and IT professionals? Simply stated, AR encompasses any experience that combines video (live or streamed) with layered objects, such as 3D animations and other useful data, to create a visually enhanced relationship between content and the viewer. AR may be stereoscopic (3D) or not, and may also allow the user to interact with the technology by tapping, swiping, movement or a wide range of other actions.

Virtually Different

How does AR differ from virtual reality (VR)? Actually, AR and VR complement and overlap each other, and are increasingly combined with each other and with video in the aptly named “mixed reality (MR).” VR specifically creates a more enveloping experience by tracking viewer movements to orient the displayed images. Some VR devices and apps are also stereoscopic, and produce totally immersive 3D experiences. VR can also allow the participant/s to impact this virtual world through the use of virtual hands, tools or user-defined gestures.

MR, AR and VR are important to AV/IT and video professionals because they create a powerful way to add value to visual content and systems, and to expand the possible ways viewers can interact with and use visual data. Increasingly, enterprise and institutional end users are looking for media designers and systems integrators who can help them use these powerful new tools to achieve their marketing, training and entertainment goals.

AR & Video

The background image-plane for AR is typically the live, local video from a smartphone camera with other visual elements, such as 3D animations or context-sensitive information, superimposed. Integrating live video streams into the AR experience can add impact in several ways. An “ordinary” AR application can, for example, add a live video stream as a PiP (picture-in-picture) window. Think of Tony Stark in Iron Man adding a video-call window of Pepper to his heads-up display. In reality, live streams can add live video support to augmented tourism, shopping or training apps.

There are many powerful ways to combine live streams with augmented content for use in training and education. A live smartphone view of the object of study, whether it’s a circuit board or an apple pie, can be combined with the instructor’s video window, and layered with relevant graphics and animations. The student/viewer can also activate keyed inserts of diagrams, animations and even live video of other remote participants.

Turning the tables, a live stream can also be used as the background, making the remote scene the reality to be augmented. This technique can enhance a live lecture, sporting event, museum visit or news report with user-relevant data, animated graphic overlays or even additional canned or live video streams. Though live, multi-stream-plus-graphics experiences may currently be beyond the budgetary or technical scope of most organizations, hyperbolic tech progress makes it likely that high school students will be deploying mind-blowing multi-stream augmented expressions within a couple of years.

Tools For T. rex

One of the simplest, yet most effective combinations of live streaming with AR is employed to create the “you are there” scenario. As far back as five years ago, INDE created a large-screen experience for National Geographic, where visitors could see themselves (and their surroundings) augmented by full-motion astronauts, dolphins and even T. rex. Today, INDE offers a variety of solutions, including the aptly named Broadcast AR, which can be deployed via canned augmentations (à la Nat Geo), or using custom 3D content from INDE’s creative team. For more info, go to

Creating AR applications using existing 3D assets should become easier with the introduction of such second-generation toolsets as Vuforia’s Enterprise Studio. These new tools are designed to minimize the amount of coding required to build apps, and reduce the complexity of incorporating 3D CAD and IoT elements into an AR experience.

Reality check: Adding live video streams to mobile AR experience is still no trivial pursuit. The usual challenges of bandwidth, latency, synchronization and buffering delays (to name just a few) are compounded by the addition of user-driven content, motion control and device compatibility. Most current development toolsets do, however, offer ways to program multi-OS and multi-device compatibility for a variety of VR headsets or smartphones.

Mixed Reality

Mixed reality combines AR, VR and local or remote video streams for a richly immersive experience.
Mixed reality combines AR, VR and local or remote video streams for a richly immersive experience.

Using a combination of streaming video, AR and VR, mixed reality offers incredible content enhancement opportunities for applications where maximum user/viewer participation will yield results, such as in training and point-of-sale digital signage. Northway Games, a pioneer in mixed reality, has produced a practical guide for developers venturing into these mostly uncharted waters. It is available for free at

High-end systems from Ross Video, Orad and others enable the creation of a more sophisticated level of mixed reality content. These systems have been developed to address the major challenges of coordinating multiple moving camera shots with static or animated 3D graphics. Although used primarily for virtual set production in television and film, they also make it possible to create ultra-realistic mixed reality environments.

Deploying an augmented or mixed reality application or experience can be done on a DIY basis using free or low-cost tools, such as Layar or Augment. Layar offers free trials, and its Creator Basic lets you deploy a simple AR experience (with video playback, if you wish), for a cost of $3.50 per page per month. Jo Ann Brezo, writing on Marxent AR Blog, discusses many of the tools available, and outlines the variables that determine AR deployment costs at “At Marxent,” she reports, “custom experiences typically start in the neighborhood of $25 to $30k.” Another good source for up-to-date info about corporate and institutional AR apps and techniques can be found at

Seventy-five million players downloaded Pokémon GO in the first month of its availability, outpacing the adoption rate of Twitter and other major online experiences. Even if there is no obvious way that you or your clients can immediately benefit from using AR, you owe it to yourself to get more familiar with this phenomenon.

Go ahead: Download Layar or one of the other free starter toolsets, or at least Google what your colleagues and competition are doing with AR. You may be pleasantly surprised, and might even find a way to augment your financial reality.

Previous ArticleNext Article
Send this to a friend