AV9000 Checklist Item Under Test: 3.7.4: “Entanglement” issues included in the needs analysis/programming phase have been accounted for, and failover processes have been documented in the narrative. Immediate actions during a failure have been considered, along with their possible solutions (e.g., redundant power supplies, system switchers, control systems, patch bays, etc.).
Reasoning: Inevitably, something will go wrong with a complex AV system. There is too much technology communicating with other technology for the system to have 100-percent up time.
For some systems, if something fails, it is acceptable for the participants to move to another room; for other systems, failure is not an option. This is the case for Entertainment Venues.
Entertainment Venue systems should be designed with failovers built into the operation. Operators should also practice these failover procedures to assure they function and, when needed, they can be implemented quickly.
The level of redundancy depends on how critical the space is to the success of the owner. It could include a spare display on wheels for simple systems, or it could include three separate ways of routing sources.
Addressing and eliminating single points of failure should be discussed during the programming phase of any project, and periodically practiced during the system’s life.
The Story: If you have ever traveled with little kids, you already understand how important redundancy is. Whenever my wife and I take “the smalls” to a restaurant, we use a three-tiered approach to maintain the peace and avoid “high-pressure situations.”
- Level One centers on crayons and coloring books. It is creative and quiet.
- Level Two revolves around their current favorite toys. Not only do toys occupy them, but they can also be used as a reward. For example, “If you have one more bite of broccoli, you can play with your Gudetama Lazy Egg.”
- Level Three…the last level, which comes out in emergencies…is the iPad. I’m not proud of it, but it works in a pinch.
So, for quick meals, coloring books are all you need. However, if it’s a busy night and the kitchen is backed up, coloring might lose its appeal; that means we have to pull out the favorite toy. If the kitchen is backed up and the place is out of chicken tenders, then not even a coloring book with stickers or a new toy will make up for the fact that we had the audacity to order our children hamburgers. In such cases, there is no choice left but the iPad. That is the triple-redundant, keep-your-child-happy Restaurant Entertainment System* the Maltese family uses. (* Patent Pending)
We see the same thing in AV systems, especially in mission-critical venues in which the show must go on. I’ve written about the triple-redundant HD-SDI MPR system that had three methods of routing sources: control system, router control panel and patch panels. I’ve also written about the trading floor systems that had standardized on a single pit design, and they kept two extra systems on standby that could be swapped in minutes if anything went wrong during operation. Those are extreme examples, but harmonics of their effectiveness can still be very useful. My question is this: Are we talking about these possibilities with our clients?
Most conference rooms are simple enough to be replaced with a user’s laptop and an HDMI cable. As long as a spare display is somewhere on the floor, either an LCD on wheels or, dare I say, a portable projector, the meeting can continue in some form with software codecs and presentations. However, as simple a solution as that is, a question must be asked: Has the support staff been trained in efficient system failover? For example, does the receptionist or AV operator know where the spare display is? Has it been turned on in the last two years? If the room requires a speakerphone, are the ports at the table configured to accept it? Even if redundancy has been built into the system, many AV owners neglect their systems currently in use. I can only imagine the backup systems are neglected to an even greater degree.
For entertainment venues, let’s assume that redundancies (we hope) were included in the design. These spaces, which most likely have dedicated AV operators who run the shows, don’t always allow the operators time to drill failover Standard Operating Procedures. The integrator who installed the system might not even have taken the backup devices out of the box.
I remember one system that included two telephone hybrids (primary and backup) that were controlled by the control and audio systems. However, the mixer that had the two hybrid modules was a single point of failure for the system; for that particular space, that was unacceptable. So, standalone telephone hybrids were included in the design and delivered by the integrator…but they were never tested.
Well, as luck would have it, one day, during an event, the DSP mixer stalled. The eager-to-please operator remembered having read about the backup hybrids, and went to find them. The phone lines were patched in, and only then was it realized that an analog phone set was needed to dial out. When was the last time you saw an analog phone set anywhere in a modern office space? The failover procedure was never tested and never documented; as such, it was useless.
Not including failover solutions in system designs is like taking your kids to an outing without a bag filled with goodies. Not drilling failover procedures for the operators is like not checking the battery of the iPad before you leave the house. Both can be disastrous, although, arguably, bringing an iPad with two-percent battery is worse. You have the three-tiered failover solution in place—it just doesn’t work at the moment. And I’m just not sure which is a higher-pressure situation: when the person in front of you having a meltdown is a toddler or a senior-level executive.