Can't we all just get along? That plea, first from the late Rodney King and later from the U.S. president played by Jack Nicholson in "Mars Attacks," is a long-sought and much-desired goal for control and automation engineers, their IT counterparts and their companies.
It's been quite a while since fieldbuses, Ethernet, Internet, wireless and other digital technologies began to encroach on the plant floor's traditional point-to-point networks for operations and control, but engineers and IT still get on each other's nerves.
Jim Montague is the executive editor for Control. Email him at [email protected].
"We used to sit in project meetings with only the controls guys, but in the past few years IT people started attending, and it's gotten better," says Keith Jones, PE, president of Prism Systems, a system integrator in Mobile, Ala. "However, we were just in a meeting with a Fortune 500 manufacturer, and its engineers told us ahead of time to listen to what IT was going say, but warned us not to agree right away to do anything. They told us to wait, and the engineers would decide if what IT wanted could be done. They had concerns, and wanted to make sure they could override any IT requests, if needed. Apparently, IT had previously pushed some Windows patches onto the plant network without notifying the controls side, and it had shut down some production lines. So the controls engineers demanded that patches couldn't be forced."
So, why didn't these controls engineers and their IT counterparts talk out these issues in the first place? Why weren't they straight with each other in their meetings? And why didn't they settle on mutually understood patching policies and project requirements before operational problems happened? Whatever the history and reasons for these continuing squabbles, many managers are sick of them, and are demanding that they stop. Fortunately, there are a bunch of new strategies and tools for dividing and coordinating network responsibilities to prevent finger-pointing and settle these old conflicts.
Start With Leadership
The primary initial element in getting controls engineers and IT to cooperate is their managers. While these conflicts often were ignored in the past, many leaders not only are requiring both sides to cooperate, but are reorganizing them under common managers, and developing hybrid engineers with both controls and IT skills.
Nagesh Nidamaluri, senior general manager at Mahindra Vehicle Manufacturers (MVML) in Mumbai, India, reports that his company's 14 greenfield auto plants and shops in 280 acres in Chakan, India, work with their IT departments to connect the manufacturing execution system (MES) on its huge and expanding plant floors to its enterprise resource planning (ERP) system.
"We're connecting all of our many varied components and production equipment with IP addresses, which gives us the flexibility to be lean and expand as needed," he explains. "This method also gives us visualization for our plant managers, so they handle production in real time, and shift or expand production lines more quickly in response to supply-chain issues and other situations."
Traditional IT vs. control engineering conflicts were resolved mostly because he now oversees both departments, and he's encouraged them to work together, Nidamaluri adds.
Figure 1: Workers assembling vehicles at Mahindra Vehicle Manufacturers' 14 plants and shops in India rely on a campus fiberoptic and Ethernet-based LAN that jointly supports VoIP, plant and IT data, business systems, security and alarms.
Source: Molex and Mahindra
To serve present production and future plans, the campus local area network (LAN) for Mahindra's Chakan project was designed and implemented to support applications such as Voice over Internet Protocol (VoIP), plant data, IT data, business systems, security and alarms. Each plant has its own network hub room, and uses a backbone of single-mode 10 Gbps fiberoptic cable between shops and multi-mode 1 Gbps fiberoptic cable within each shop, which enables dual-path redundancy, rerouting and self-healing during recovery. The body shops also use Cat. 6 UTP cable with IP67 bayonet jacks to withstand vibration and protect against dust, water and oil (Figure 1). So far, system integrator Wipro Technologies and Molex report they've helped install 165 km of UTP/FTP cables and 124 km of OFC cables at the shops, and have reserved space and capacity for planned wireless devices in the future. This backbone also assists Chakan's green initiative and sustainability systems, such as heat recovery, solar panels, water treatment and several manufacturing processes.Divide and OrganizeAs important as leadership is to motivate cooperation, there are practical and technical obstacles. These can be overcome with logic, organization and prioritization. It begins with the best way to divvy up an industrial network, which is to partition it into logical, functional subnetworks, and then separate and isolate these subnets with managed Ethernet switches and/or firewalls. "Good fences make good neighbors," says the neighbor in Robert Frost's poem, "Mending Wall," and this is especially true for today's industrial networks.Of course, sorting out all the process applications and network systems they need is probably a little easier if much of their equipment isn't out in the middle of the sea. This was the challenge faced by system integrator Cimation of Metairie, La. In 2010, its team was asked to implement an extensive Ethernet-based network, including automation, supervisory control and data acquisition (SCADA), cybersecurity and business functions for a fixed oil and gas processing platform in 2,000 ft of water in the Gulf of Mexico. The platform was previously operating in a limited capacity, but the owner embarked on a capital improvements program to greatly increase its capacity. Because the enhanced platform design would process a high volume of about 30,000 barrels of oil equivalents per day, the owner wanted the platform network to have maximum reliability and uptime. Also, the owner asked Cimation to model and pre-prove the network and wireless communications on shore, and then install them without disrupting the platform's existing operations.
The platform serves multiple wells that feed the facility via pipelines that run along the seabed, and has processes that separate oil from natural gas. The oil is pumped and the gas is compressed into pipelines that deliver these products to facilities onshore. The network requires 10 servers that poll data from PLCs and other automation devices, provide security services such as closed-circuit TV, and manage domain controllers for credentialing and login. This system also includes terminal servers that provide HMIs; historian servers for archiving time-stamped production and alarm data; and reporting services that connect to a large enterprise network via a microwave link to offices on shore. Another stack of servers talks to subsea systems via the OPC protocol. They perform proprietary flow calculations on the composition and volumetric flow rate of oil and gas, a custody transfer function. Finally, some of the platform's controllers use UDP multicast messages, which creates traffic that has to be filtered to protect certain control devices.
"Older or smaller platforms have PLCs and HMIs locally networked, but they may not have any business subnet, security services, or communication with subsea systems. Those types of facilities sometimes only report periodically via a modem, so their capabilities are limited, and the smaller quantity of data provided to the enterprise is just a periodic summary," says J.D. Bamford, PE, CRM/SCADA security engineer at Cimation. "To provide continuous remote monitoring, a full business subnet, management and security functions, the new network needed firewalls and demilitarized zones (DMZs) to segregate these portions of the network. It also needed the capability to continuously broadcast operations to a backup control room located at the offices on shore." This backup capability could be used if the platform crew is forced to evacuate during a hurricane.
Any network that spans business and automation areas requires a service-level agreement between operations and IT based on what applications are in the field, and what the enterprise network needs from those applications. Cimation interfaced with enterprise data users as well as operations personnel to define a data exchange strategy that dictates the type and direction of data flow between the platform and shore.
Figure 2: This simplified diagram of the network that Cimation designed and helped implement on the oil and gas platform in the Gulf of Mexico isolates process control and business layers of the network using routers and firewalls, and permits only the minimum traffic necessary between these layers.
Source: Cimation and Belden
Bamford explains that this data exchange strategy was part of a broader effort by Cimation to architect the platform's network using a defense-in-depth cybersecurity strategy. The design followed the IEC/ISA-62443 security standard (formerly known as the ANSI/ISA-99 standard), along with guidelines from the U.S. Department of Homeland Security and US-CERT. These guidelines provided a framework for Cimation's effort in partitioning the network and isolating the process control and business layers using firewalls, DMZs, antivirus services, and related measures. Cimation installed a dozen Tofino Security Appliances (TSAs) to act as firewalls. These DIN-rail-mounted appliances weed out unauthorized communications, manage traffic between automation PLCs, and protect vulnerable platform controllers from excessive traffic and unwanted intrusion (Figure 2). The TSAs with Loadable Security Module (LSM) software were installed in front of Rockwell Automation's ControlLogix PLCs, for example, and are managed by the Tofino Central Management Platform (CMP) from Tofino Security."The biggest challenge was that while we could bench test some of the network, its communications and the main automation PLCs, there were scores more devices, which contractors and third parties would be installing on the platform concurrently," Bamford explains. On such a large project with so many parties contributing equipment, it is inevitable that some won't provide their data exchange guidelines in time to be designed into the system. "So when some vendors were told where to plug in, they found that they couldn't communicate through the firewall. We met with each of them (along with the end user), validated their devices and functions, approved a firewall reconfiguration, and then tested to make sure the appropriate messages got through. This was after we'd already done as much preplanning and pre-configuration as we could, and surveyed the vendors on what protocols they were using and what ports they needed open. The primary and backup process and safety PLCs were all bench tested, but the switchgear, subsea cabinets and other process packages had to be integrated in the field."
New Roles and Responsibilities
Naturally, all these new methods of managing networks are going to change a few job descriptions. Prism helped implement a greenfield MES and plant control network last year on CNC and assembly machines at a valve manufacturer in Guadalajara, Mexico, and the IT staff installed the whole network and all the components down to the PLCs. "They were fully involved, configured the switches, and put in the runs to all the equipment," says John Elias, Prism's network manager. "The machines run on a private EtherNet/IP subnet, so they're in their own broadcast domain and set their own IP addresses, and then route data up to the main network by creating a bridge with managed Ethernet switches, setting up a virtual local area network (VLAN), or using network interface cards (NICs). It's easy to put multiple NICs in a PLC rack, such as one for local I/O and the other to communicate with higher-level networks."
As formerly separate network lines merged, it became harder to say where the IT staff's responsibilities end and where the control engineers' jurisdiction begins, agrees Jeff Payne, product manager for PLCs, I/O and PCs at AutomationDirect. "It's everyone's responsibility," he says. "We can't say industrial networking is just the job of one person or another. And this joint responsibility also must be considered when networks are being designed."
Tools Aid Tolerance
Though it's crucial for controls and IT professionals to talk and take on new tasks, many technical aids have popped up that simplify network design, implementation and maintenance — and smooth out former network jurisdiction snags.
For instance, subnets teach some users to explore the added flexibility that IP addressing gives their Layer 3 and multilayer managed Ethernet switches and VLANs. Where Layer 2 switches use one IP address on a flat network, newer Layer 3 switches and their routers enable communication between IP addresses, and this lets them serve as firewalls, sort out authorized messages and data, and enforce patching policies.
"Plant guys don't want anything other than their industrial communications to come through their firewall, and Layer 3 switches let them do it more easily," says Jim Toepper, marketing manager for industrial networking and video solutions at Moxa. "Layer 3 switches have been in our industry just three or four years, but they're growing at about 180% per year."
Likewise, Prism reports that the Common Industrial Protocol (CIP)-based EtherNet/IP standard didn't previously support other types of generic Ethernet's TCP/IP, but this year's Version 20 and 21 firmware releases will talk to regular TCP/IP. Other protocols, notably Modbus TCP and Profinet, already communicate with standard Ethernet.
"Over time, the initial barriers to Ethernet's use in plants came down as managed switches threw more pure speed at them," says Mike Miclot, marking vice president for Belden Americas. "But this still isn't as simple as adding a particular cable. Developers must use some forethought when designing these networks."
Security Coaxes Cooperation
Though it's sort of a chicken-and-egg situation, one of the main forces driving controls engineers and IT to settle network arguments is a frantic need to secure all the multiplying networks and endless links they established, even as more are created. Unauthorized probes, intrusions, hacks and malicious software scare everyone, and are a wakeup call for many of them. Nothing inspires cooperation like a common enemy.
Luckily, many engineers and senior managers realize the importance of cybersecurity in automation, and support using IT-based security tools such as patch management policies, antivirus software, event management, network segmentation and firewalls.
However, differences in perspective persist between controls and IT. The highest network-security priority for process control engineers is preserving availability, while the highest network-security priority for IT technicians is protecting confidentiality. Implementing patches and antivirus software is relatively easy for IT systems, but it's difficult for plant-floor networks that can't shut down as easily. Seeing beyond personal differences in outlook on the way to cooperation is even trickier.
"We're pulling more data from the plants to our OSI PI systems, and there are more remote connections, too. More connections means more vulnerabilities," explains Gregory Rogers, PE, senior manager for control engineering at Enterprise Products in Houston, which provides mid-stream processing, storage and distribution services for U.S. oil and gas producers in 38 states. To shut its backdoors, plug holes and fix vulnerabilities, Enterprise white-lists its systems and facilities, performs regular patch management and antivirus software updates, follows the ISA 99 standard's recommendations for minimizing and managing pathways on its networks, and creates segmented zones for different functional areas.
Besides all its technical fixes, Enterprise also secured management's support and involved its entire staff on network security. However, challenges remained, even after everyone was on the same page. "When we did cybersecurity at Enterprise, we held a WebEx meeting that was attended by several hundred people," Rogers says. "Even so, our fractionalization plant in Mount Belvieu has 50–70 PLCs, and 16 of them were connected to the corporate network and had added Ethernet cards to provide access for troubleshooting. One day, a guy was conducting a training class in a conference room, and so he took a hub and the class plugged into the network. Later, someone plugged that hub back into itself, which created a broadcast storm that shut down three of the plant's PLCs, and brought the facility to its knees. It was fixed the next day, but it was still pretty embarrassing."
Personnel-based security must seek to create a culture among the control engineers of cybersecure thinking. It also helps to rope in corporate IT professionals, who can pass along critical thinking about control systems — once they know what's critical to the controls side. "Training and increasing staff expertise on cybersecurity will minimize events within the ICS domain," Rogers explains. "We used to move USB sticks between stations, but we don't anymore. We don't just put hubs on the network anymore. Now, we can use FTP sites or relay servers, and transfer data through a DMZ, so the antivirus software can catch any problems. We have a cybersecurity committee, too. So, when the U.S. Dept. of Homeland Security puts out cybersecurity threat notices about eight or 10 times per week, we have a group that can advise on whether this is a real threat to us."
Prism's Jones adds, "The good news is that controls engineers have been working more at the Ethernet level, so they understand and appreciate more of what the IT guys do, and they're also meeting with them and showing them how Profinet and EtherNet/IP are different than TCP/IP. As a system integrator, we frequently cooperate with IT to get static IP addresses for passing data to the enterprise. Now it's rare that we don't talk to both controls and IT. They're both understaffed, and so they're realizing it's easier to work together. With all the downsizing and fewer resources these days, cooperation is the only way to get jobs done."
