Deep underground, within a system of mountain tunnels connecting the final leg and a particularly hazardous section of Colorado’s Interstate 70, is one of the most technologically advanced command and control centers in the transportation industry. The tunnels lie within Glenwood Canyon, an extremely narrow, environmentally sensitive and spectacularly beautiful 12.5-mile road connecting bridges, viaducts and tunnels. The command and control facility is located within the Hanging Lake Tunnels, a 4000-foot, two-bore tunnel through the mountainside in the canyon.
Design, engineering and integration firm Novate Solutions recently completed an upgrade and expansion of the command and control facility. The multiyear project encompassed the control system modernization and replacement of network infrastructure covering more than 30 miles of partially elevated highway and tunnels. The expansion included the installation of a videowall and connecting the control to the existing analog video system.
In addition to being a critical and direct conduit of transportation through the state, the multi-award-winning Glenwood Canyon project was also recognized as the “last link,” completing the original US interstate highway system when the tunnel opened in 1992.
The tunnel’s command and control facility was first established in 1991, just prior to the Glenwood Canyon project’s completion, to provide manned 24/7 oversight and first-responder capability to this remote stretch of road frequently beset by rockslides and heavy snowfall. In 2012, Novate Solutions was engaged to update the control system that, back then, consisted of an antiquated combination of TVs, cameras and monitors.
In that initial project, Novate overhauled the control system, deploying an utterly failsafe integrated Distributed Control System (DCS) designed to provide better situational awareness and visual detail to the facility’s command and control operators, as well as its first responders.
Spotting and responding to roughly 450 incidents each year, ranging from simple fender benders to catastrophic fuel spills or rockslides, puts a significant strain on personnel. Not only are lives at stake, but a road closure means a 3.5-hour detour, or being stuck on the road in dangerous weather conditions. The remote location of the tunnel requires crews to live onsite during their shifts, working three days on and three days off.
The facility had to provide failsafe operation in a highly dynamic environment, seamless communication between all devices and built-in redundancy and reliability without interruption of operation.
The product selection criteria was based on the requirement to switch from proprietary equipment that was no longer being supported to off-the-shelf process automation equipment that uses Common Industrial Protocol (CIP). Equipment had to be robust, highly secure and easily interoperable with other devices. These criteria specifically drove the use of Cisco and Rockwell equipment, both of which are well-established industrial brands that use CIP with proven track records for robustness and interoperability.
A significant portion of the equipment budget was spent on Jupiter Systems [now InFocus], given that the project was about visualization and the implementation of a videowall. The use of InFocus/Jupiter’s solutions was based around scalability and fault tolerance. InFocus/Jupiter PixelNet was the only solution on the market at the time that would support multiple failures and result in no more than a 20% reduction in performance. InFocus/Jupiter’s Canvas software offers security features that were also instrumental in the overall package selection of InFocus/Jupiter as a key equipment vendor for this project.
Integrated information and visual detail helps the control room operators more accurately monitor incidents and assess the proper response, thereby improving response times and reducing operator stress. The DCS system network infrastructure uses more than 80km of fiber (OM1and SM1) covering thousands of I/O points that are
monitored and controlled on a 10-millisecond basis.
With this backbone in place, Novate installed a 27’x9′ seamless digital wall of 28 Planar Matrix 55-inch remote-power 50,000 lumen screens to replace the individual computer monitors. The videowall, driven by a network of PixelNet video input and output nodes from InFocus/Jupiter, integrates live camera feeds from an analog video system with the DCS. PixelNet delivers high-resolution, real-time video from any camera or data source to any screen. Any information source can be shown on any display as a window on a single display, or as a window spanning multiple display devices in a display wall.
The Hanging Lake Tunnels video-wall allows operators to view multiple locations from multiple video feeds simultaneously, as well as to instantly identify the exact location of any spotted incident for precise communication to first responders. The DCS also enables operators to broadcast text to the 42 roadside message signs and 82 lane-use signal signs (indicating open and closed lanes) when safety warrants such notices.
CDOT Office of Information and Technology Services Lead Kim DeCoursey explained that, although the facility’s primary focus is on life safety control and unobstructed traffic flow, the department also has a responsibility to Colorado taxpayers. “The upgrade, completed five years ago, cost effectively shifted the technology from a proprietary to an open architecture. Simultaneously, each new system component was evaluated based on reliability and cost-effective performance. The resulting infrastructure and integrated system makes it possible to continue upgrading the command and control system over time to incorporate new and more powerful devices to enhance public safety without issue or interruption of service.”
Hanging Lake Tunnels set the bar for technology management of tunnel traffic five years ago and has maintained that status through this flexible, future-friendly infrastructure.
Novate is now engaged in a comprehensive update of that equipment and technology designed to further improve response times through more granular/visual data views, while simultaneously adding two-way communication flow and the ability to selectively communicate information to the public and other CDOT facilities.
Novate is switching all analog feeds to IP, adding a multi-honed 1Gbps-speed network on top of 25-year-old OM1 62.5/125µm fiber originally designed to support only 256kbps. The network carries fire and safety monitoring (CO levels, heat detection, etc.), operation of fire suppression pumps, 2400HP fan operation and traffic control data.
Analog cameras are being replaced with streaming high-definition cameras, along with FLIR thermal imaging cameras providing both visual and thermal data. Thermal cameras will replace obsolete linear heat-detection systems and will set trip points to notify the control room of high heat or fire. When this happens, the videowall automatically changes to a fire layout displaying video images from the FLIR cameras. All images are produced and delivered to the DCS through Rockwell/Allen-Bradley Process Automation Controllers (over Common Industrial Protocol) while securely pushing the H.264 stream through a multiple-failure-point-tolerant, Cisco IE4000-based, fully quad-redundant fiber-ring IP network. From the top of this network, video data is passed through a pair of redundant Cisco 3850XS Layer 3 switches to a core based on a Vivotek VMS system.
Thermal, HD Cameras
The thermal and HD cameras significantly improve the quality of visual information provided to facility command and control operators, along with more precise coordinates for locating the incident. “Enhancing the visual acuity of our cameras, along with streaming views, enables us to more quickly spot a pending incident and helps us respond with the right equipment at the right place,” said Todd Anselman, Hanging Lake Tunnels Electronic Specialist IV. “We have 33 extraordinarily dedicated operators here, and in so many situations, seconds matter. The more information we can provide to augment their significant emergency response experience, the better service we can provide to the public and, in turn, reduce stress in our personnel in quick-response situations.”
Facility management also wanted support for bidirectional video feeds so first responders could see a broader spectrum of visual information on the incident and, in turn, transmit additional incident information back to the videowall from their phone or onboard truck computer. In addition, facility management wanted to be able to share and continuously provide selected video feeds to remote CDOT offices for assistance or collaboration.
To accomplish this video sharing, Novate installed Canvas collaborative visualization software from InFocus/Jupiter. A Vivotek VMS system manages video streams from both the legacy analog cameras and the high-resolution cameras.
Canvas accepts both H.264 and H.265 video feeds from cameras and analog to H.264 encoders, accessing both live and recorded content. Canvas streams HD video from the secure internal network to remotely located control rooms where disaster mitigation teams can monitor or assist as necessary. In the near future, Canvas streams will be pushed through a Verizon private network to secure authenticated mobile clients for on-the-highway, real-time viewing of fire, accident and weather situations.
Canvas data is routed to PixelNet for display on the videowall. The DCS system continually monitors the health of all components, including InFocus/Jupiter hardware and software, Cisco equipment, Vivotek and all other process automation components, logging data and audibly announcing any issues to the operators with simultaneous SMS notifications to technical staff.
Transmitting data through a mountain of rock in near real time is challenging. Simultaneous broadcasting of bidirectional streams of multiple data feeds with continual monitoring of every component within this system is enormously complex. Above all, the entire system must be robust and completely failsafe. This system was designed to survive the remote possibility of multiple point failure from any point in the system, from field I/O to video management systems. Single switch failures could not be allowed to impact the system and, further, the system must be able to tolerate an entire bore failure (tunnel fires will melt fiber) without any data loss.
Accordingly, all critical components, such as the Cisco 3850XS switches, the PixelNet Domain Control servers, the PixelNet video nodes, PixelNet switches and Canvas servers are all redundant, so any failure will result in minimal loss of video and data streams, if any. Further, the system must be able to be maintained by field staff alone for 24/7/365 operation. Rockslides and heavy snow conditions will sometimes isolate command and control operators for significant periods of time. In addition, any issue has to be reported audibly in the control room, visually in the DCS and via SMS to technical staff.
“The Novate team has in-depth, long-term knowledge of this facility,” said DeCoursey. “Not only do they know the systems here, but they understand where technology is going and they keep us on top of our game. They consistently exceed our expectations and meet significant challenges in a practical and cost-effective way, with an overriding focus on public safety.
“Glenwood Canyon will always face significant environmental and weather issues. The Hanging Lake Tunnels command and control facilities must keep pace with both what nature and drivers throw at us every day to keep this breathtaking place accessible for travel, recreation and transit.”