Software Problems: A Glitch In The Matrix?

Checklist Item Under Test: 6.2.11: The speech reinforcement system is stable (no feedback) for the entire talker and listener areas specified.

Reasoning: With decentralized, open-architecture DSP mixers becoming more and more popular, you would think that troubleshooting would get easier. The audio cable runs are shorter. The audio gets converted to digital data immediately. The entire audio system can be accessed, reviewed and troubleshot from an easy-to-customize-and-navigate GUI. However, that GUI doesn’t tell you everything, especially when the issue is most likely a compiling issue in the software. These can be quite difficult and time-consuming problems to hunt down. This story details my troubleshooting experience with one such challenge.

The Story: How creepy was it in The Matrix when Neo saw that black cat walk across the doorway for the second time and give a little shake?!

Neo: Déjà vu.

Trinity: What did you just say?

Neo: Nothing. Just had a little déjà vu.

Trinity: What did you see?

Neo: A black cat went past us…and then another that looked just like it….

Trinity: (speaking urgently now) How much like it? Was it the same cat?

Neo: It might have been. I’m not sure.

Morpheus: Switch! Apoch!

Neo: Why? What is it?

Trinity: A déjà vu is usually a glitch in the matrix. It happens when they change something.

I recently came across an issue that I can only attribute to such a glitch. I’ll share the issue with you and the steps I took to attempt to solve it.

The client complained about audio feedback and echo from a system that had been in operation for years. They recently made some minor control changes, but nothing in the audio parts of the site file. The audio anomaly sounded like a cross between feedback and acoustic echo, but it would happen with local audio, and only in the low frequencies. The user actually localized it to one of 24 wireless microphones in the system, in that when they muted Mic 8 in the control system, the issue cleared up. (Note: The issue was present when the transmitter was powered down, but the channel itself was unmuted.) My job was to get rid of that anomaly.

First, I wanted to see if it was an issue with the wireless microphone receiver channel. I swapped wireless microphone channels by swapping Mic 7 for Mic 8 and vice versa. The issue stayed on the Mic 8 input of the mixer, and the actual eighth wireless microphone sounded great on the input for Mic 7.

Second, just for the goof, I wanted to make sure there was nothing wrong with the cabling. Maybe there was a loose conductor that was picking up some weird induced feedback somewhere. I unplugged the cable at the microphone receiver and the issue was still there. I then unplugged the cable at the mixer input…so nothing was connected to that Mic 8 input of the mixer…and the noise was still there.

This installation used a decentralized DSP mixer, so there were inputs and outputs spread out all over the conference center. I figured maybe there was something wrong with this particular input card module that brought Mic 8 into the system. I powered down the input module, essentially removing it from the signal chain. It showed as “unavailable” on the DSP GUI so I knew I had the correct module. However, the noise was still there.

At this point, I know the issue is not the microphone receiver. It is not the cable. It can’t really be the input module because the issue is still there when the module is powered down. It has to be something in the software.

When I tell you I looked in every nook and cranny of the site file for something different between Mic 8 and its other wireless microphone brethren, I mean I violated that site file. I felt bad at how brutally I looked into every single checkbox and setting available. It was the DSP equivalent of a full-cavity search. I saw no difference between the microphones. The settings were exactly the same.

Now I’m thinking that there must be something wrong with the main processor, but this processor was running eight other rooms perfectly well. At this point, I was at a loss. So, I did what any self-respecting AV specialist would do. I asked the client if they really needed all 24 wireless microphones. As it turns out, they didn’t. Consequently, I deleted the “connector” in the site file between the microphone input and the rest of the DSP modules, and it worked like a charm. There was no more noise in the system. I didn’t fix the system, but I did solve the problem.

If I had unlimited resources, time and specialists with me, I would have factory reset the device, reloaded firmware and tried loading the site file again with the problematic Mic 8 connected. Maybe getting the processor to reload the site file from scratch would have cleared it up. However, as I mentioned, the processor supported eight other rooms, which would all have to be recommissioned after a step that drastic. The client said they could live with 23 wireless microphones. Everyone was happy except me. I hate when I can’t fix things. If had to guess at the root of the problem, I think there was a compiling error somewhere that did something strange to the audio chain that brought Mic 8 to the party. Now I’ll never know.

Reader, I know you were probably hoping that, like Neo, I started to visualize the audio in the space in its native binary state flowing throughout the conference room and found that I could take control of the immense amount of data and cause the noise to cease simply by willing it to stop. If I had The Wachowskis on speed dial and a couple of million dollars for CGI, I would love to share a scene like that with you. However, we live in the AV world, and not every story can end with satori. Sometimes it’s OK for clients to survive with only 23 wireless microphones.

Previous ArticleNext Article
Send this to a friend