The trick, as a kid, to procuring alcohol, especially clear alcohols like gin, vodka and even Sambuca, was to replace whatever was “borrowed” from the bottle with water. Even if your parents marked the bottle to keep track of how much was left, by refilling it with water, who would be the wiser? You would be the coolest kid in school for bringing half a mug of vodka to the party, and your parents would just have a slightly watered down cocktail for a bit. This plan was working out brilliantly for us, until one of my friends got caught. His parents, unlike ours, would keep their liquor in the freezer. The alcohol content of most liquors is high enough to prevent the liquid from freezing. This is typically the case, unless, of course, half that liquor is replaced with water. Then the bottle freezes solid. And the parents start to get suspicious. Naturally, reminiscing about this got me thinking of AV.
We were commissioning a large MPR system that had an IP-based video routing system. Each transmitter and receiver had a statically set IP address. Since the router supported a true multi-purpose space, not all the transmitters and receivers would be connected at the same time. If a laptop wasn’t required at Floor Box B, its transmitter was not connected to the system. After a few hours of testing, we noticed several transmitters no longer operating with the system.
It was a fairly new product, so, naturally, fingers were pointed in the direction of the manufacturer. However, with some manufacturer support, the problem was traced down to IP address conflicts. All system endpoints in the video router, and other AV devices, all had static IP addresses to remove the risk of conflicts. Despite this, if an endpoint was left unplugged, that IP address was up for grabs by the system DHCP router. Typically, the network engineer can assign a range of addresses for DHCP to stay clear of the static IP addresses, but this was not happening. And here’s where the story gets tricky.
As it turns out, the network was being managed by a router module in a control system processor. It supported DHCP addressing (and static addressing, of course). However, because it was a control system processor first, with routing as a feature, it lacked the network management tools that a full-blown managed router has. As it turns out, there was no way to limit DHCP addressing to a certain range of IP addresses. It was always on. If a transmitter with a static IP address was unplugged, and a technician plugged in their laptop set to DHCP, that transmitter’s address would be given to the laptop, even though it should have been reserved for that transmitter: the classic “move your feet, lose your seat.”
This should only have been a problem with technician laptops, which could be managed. The IP conflicts kept on happening, however. Anyone who works with large systems can attest to how fluid these large systems can be. Devices are added and removed almost daily. That fluidity was wreaking havoc. As an example, the client wanted to link all the wireless microphone receivers on the floor together via the AV network to better manage frequencies between all the receivers. The technician tasked with that was not aware how critical static addressing was to the operation of the system, and left 10 wireless receivers on DHCP because “the manufacturer had a rock-solid auto discovery feature” and “why not?” Those 10 wireless microphone receivers stole the IP addresses of 10 video transmitters on the network and prevented them from working because of IP conflicts.
One solution was to put the video router endpoints on their own subnet. This solution was met with groans from the integrator because it is not that fun to re-address hundreds of video endpoints, and then re-commission the entire system. Another solution would be to put the system on a proper managed network switch, but that was also very labor intensive. The solution that was decided upon was a good one, but I don’t think anyone felt right about it.
The control system router could not limit the DHCP range of addresses. It could also not turn DHCP off for a variety of reasons. However, individual IP addresses could be reserved. So, a text file was created reserving the 250-plus static IP addresses used by the current, known system endpoints. It worked. The IP addresses required for the video router were reserved and effectively taken out of the DHCP pool of addresses. It took a painstaking couple of hours, but required no new equipment or cabling. The integrator worked with what they had, and made the best of it.
Lessons learned from this experience:
- Just because the bottle says vodka, doesn’t mean it’s actually vodka.
If I were designing this system, and the spec sheet on the control processor said it included managed switch capabilities, I wouldn’t think twice about it. It’s not until you actually get it into production that you realized not all routers are created equal and the drastic effects those limitations can have on a system.
- Don’t trust frozen vodka. It should remain fluid even if it’s in the freezer.
A system is a living, breathing entity. Just because the design is complete, the installation is finished and people are using the system, don’t think that the system is frozen. Users will continue to change, add and remove features from the system. If there are complicated rules to make sure the system works (i.e., any device on the AV network must not only have a static IP address, but also be part of the “reserved” IP address list on the control system router), they must be made explicitly clear and written EVERYWHERE…like taped to the switches and the routers. Funny signs all over the racks are better to deal with than a non-working system.
- Stealing liquor doesn’t make you a bad person, right?
In this fluid world we live in, adding a router feature to a control system processor makes a ton of sense. There are also a lot of good reasons to offer scaled-down router features. There’s absolutely nothing wrong with a DHCP router stealing currently “unused” IP addresses to make efficient use of the network. BUT! Engineers should have more say in the marketing materials to avoid these surprises (see Lesson 1).