I was recently involved in a project where a single audio DSP mixer resided on five different networks:
- AVB-Primary for DSP mixer to DSP mixer audio signals (bussing)
- AVB-Secondary for DSP mixer to DSP mixer audio signals (bussing) failover
- Dante-Primary for DSP mixer to mixing console audio signals
- Dante-Secondary for DSP mixer to mixing console audio signals failover
- Control Network for DSP mixer to control processor communications
Not only were there three different types of networks, but the Dante and AVB networks were redundant. To maintain integrity and keep things simple in the network architecture, there was a router for each audio network, as well as a port to the client’s LAN for the control network. Everything worked beautifully. It was a mission critical system, so the redundant audio networks made sense. It was a great system.
Two months later, we got a call that there was no audio in the room. When we arrived, we found an AVB cable plugged into the Dante router. We found a Dante-Primary cable plugged into the Dante-Secondary router. Obviously this would cause all sorts of problems. AVB doesn’t speak Dante, so that path was killed. Also, the backup network ports from a device cannot be on the same network. The problem was easy enough to fix, and clearly it was operator error.
After digging a bit further, we found out what happened. There was a problem with the DSP mixer itself that a simple reboot would have solved, but an operator decided to start troubleshooting at the audio networks instead. Part of that troubleshooting including testing the cabling since there was some construction work being done in the space and they wanted to rule out a damaged cable. When the cable was put back after being tested, since there were four routers in the rack, the operator inadvertently plugged the cable into the wrong router. It’s a simple enough mistake.
In the quality management world, any issue should have a corrective action (fix it) and a preventive action (make sure it doesn’t happen again). In this case, the corrective action was to break out the drawings and make sure the cables were connected to the equipment properly. The preventive action was a bit more difficult: How do you prevent an operator from mistakenly plugging in cables into the wrong router, while they are under pressure from the users to get the system up and running as quickly as possible? Maybe you could get more complicated labels, but they would be hard to read. Maybe you can post the drawings in the rack, but they would still have to think to read them. Then it hit me. Color coding.
I’ve put together enough children’s toys to know that if you want to make sure a particular piece lands at a certain spot—regardless of stress, native language or poor vision—use color! The red cables go to the red router. How easy is that?! There are plenty of reputable sources of colored category cable or path cables. That would make connecting the devices a snap. Further, if something is plugged in at the wrong router, it would stick out like a pregnant Rockette. Color-coded category cables for systems with multiple networks would make connectivity “stupid simple.” And I jump at any chance I get to make something “stupid simple.”