I usually feel confident troubleshooting AV systems. I take my time. I make note of each step taken. I think it’s important to look at the effect of every small tweak so that the eventual solution doesn’t get glossed over in a hurricane of twisting knobs and enabling checkboxes. Not only that, but if I find a fix, I like to undo it to make sure it wasn’t just a coincidence. It becomes way more difficult when you must rely on external parties as part of your troubleshooting team. Not only is there a trust element (“Did he really just cycle power, or did he just SAY he cycled power?”), but there’s also the fact that those external elements are completely out of your control. You might not be able to adjust them. You might not completely understand them. Basically, they muddy the troubleshooting waters.
As an example, I’ll take you on a little troubleshooting journey I went on not too long ago. The client complained that during bridged audio calls, there was no continuity in the levels of the far ends. Some cell phone participants would be blaringly loud. Some desk phone participants would be quiet as a mouse. It was not an exceptional experience for the room users.
My initial reaction was for the client to lean on the bridge service to get more aggressive with their AGC. They have access to all the incoming calls, so it makes sense for them to properly adjust the levels. Further, automatically controlling levels is a pretty standard feature for bridge services. This issue with this client is that their calls have over 2000 participants (most endpoints set to listen only, but not all) and their selected bridge service, for whatever reason, doesn’t offer AGC for those conferences.
My next step was to use compression in the mixer to compress the heck out of the incoming audio call channels and then increase the levels to maintain unity gain through the signal chain. This does affect the audio quality a little bit, but the client was using a POTS line which only has 3kHz bandwidth anyway. If the levels were compressed, the users wouldn’t notice a difference in quality in the room. They already have trouble distinguishing between “esses” and “effs” by nature of the phone line bandwidth.
So, we made the simple change, dialed into a bridge, and lo and behold, it sounded great. There was very little difference in level when I yelled into my phone, and when I whispered. I was kind of feeling proud of myself, if I’m being honest. We then wanted to perform an A-B test, so we disabled the compression and went through the same tests of yelling and whispering.
It behaved the same way without the compression. Yells and whispers sounded the same.
We then added a cell phone call to the bridge. It was very loud in the room. For a second. Then it dropped down to the same level as the other call.
Almost as if the gain was automatically adjusted.
I asked the client for further clarification at this point, because I didn’t know what was going on. Why would we be performing this testing if all the far ends were the same level no matter what we did?!
As it turns out, the client uses two different bridge services. The service for the large calls (2,000+) does not have AGC, as discussed, and requires advanced notice to set up a call. The service used for smaller conferences (less than 50 participants) has an extraordinary AGC algorithm, apparently, and can be used ad hoc. We were barking up the wrong tree. I didn’t think to confirm that the bridge we were testing with was indeed the problematic bridge. I just assumed that either they only use one service, or that it would be obvious to use the one with the issues. I was wrong.
I took away two things from this experience:
- Falsifying something is good practice. Just because a problem stops after you try something doesn’t mean you fixed it. It’s also important to undo what you did to make sure the problem returns. If doing something stops the problem, and undoing that same something starts the problem again, then you’ve found it! Correlation does not imply causation.
- Don’t trust what you can’t control. If you expect something from an external component, it is good practice to confirm it…no matter how simple and obvious it is. Hence, the first two questions from any tech support operator: 1. Is it plugged in? 2. Did you try rebooting it?
I hope you found this blog helpful, and maybe even comforting…maybe even like a bridge over troubleshot waters. I hope it eased your mind.