Jim Montague is the executive editor for Control, Control Design and Industrial Networking. Email him at [email protected].
None of these needs are very difficult to fulfill. All they require is a little awareness and the initiative to go out and get them. Unfortunately, despite their many common needs, industrial networks have grown up quickly in all kinds of different applications and industries, so knowledge of their basic needs and providing for them is often lacking. Thankfully, many new methods and tools, usually from veteran, IT-based networks and their users, are showing up to help their industrial counterparts.
[sidebar id="4"]
Can't find the problem
Israel-based N.R. Spuntech Industries manufactures non-woven, hydro-entangled, Spunlace fabric used in the medical, construction, agricultural and consumer products industries. The production line at its new, 20,000-m2 plant in Roxboro, N.C., runs seven PLCs and about 350 nodes on a Profibus network. The line can turn any combination of raw cotton, cotton, viscose, polyester or polypropylene fibers into rolls of finished fabrics via a complex and tightly monitored series of operations, including opening, cording, drying and packaging.
Also read: What manufacturing facilities need to know about network security
Though the facility was new, the production line's network began experiencing random and sporadic instances of nodes dropping on and off. "Each occurrence cut the lines of communication between the line's monitoring station and critical pieces of equipment, leaving us without eyes or ears on the production floor," says Paul Hall, process control and IT manager at the Roxboro plant. "Because of these intermittent issues, we were forced to stop the production line multiple times, sometimes for hours. The resulting costs in lost revenue were substantial."
Hall was further frustrated that these interruptions in the network's communications happened without warning, without any pattern and often self-corrected without providing any clues as to what triggered them. "We were unable to physically or programmatically determine the root of the problem, and so we had to approach each new day ready for anything," Hall says. "This unresolved problem felt like a ticking time bomb and caused us a lot of stress."
Figure 1: Spuntech Industries uses Th Link central network access points to secure OPC compatibility, monitor traffic in real time and troubleshoot the network that manages the non-woven textile production line at its new facility in Roxboro, N.C.
Source: Trebing + Himstedt
Get inside, look around
To cure his sick network, Hall researched various monitoring and diagnostic solutions that could provide accessible and in-depth visibility into its performance and eventually selected Th Link central network access point components and Th Scope diagnostic software from Trebing + Himstedt. Th Link provided immediate OPC compatibility, which allowed Hall to tie the new access points directly to his existing SCADA system and monitor the real-time health of his network (Figure 1). It also enabled his operators and technicians to centralize their activities on one system instead of jumping from PC to PC to monitor and diagnose network issues.
Within the first week of adding Th Link and Th Scope to the Roxboro plant's network, Hall used it to monitor the next network interruption when it occurred, and identified and isolated a particular unit on the production line that caused the chronic node dropouts and downtime. "When this problem occurred before, we could only guess at the cause," Hall explains.
"It could have been anything from a faulty part to a loose connection. Now, analysis from our access points and diagnostic software was fast, accurate and detailed, and quickly pinpointed the cause of the interruptions. They're almost like having a Profibus technician in a little gray box. They also paid for themselves in the first week because they enabled us to make the necessary adjustments and repairs, and we haven't had any downtime related to this issue since then, which saves a lot of revenue." Hall adds that Spuntech also plans to use Th Link and Th Scope to help identify predictive trends, which will enable the Roxboro plant to be more proactive in managing its network and related assets.
Deep diagnostics — automatically
Just as medicine advanced from simple X-rays to higher-resolution CAT and PET scans, IT-based software and other tools for examining industrial networks, operations data and the content they transmit have grown more sophisticated and varied in recent years.
"The other change is that network health has gone from looking for the causes of individual problems and solving them reactively to continuously monitoring networks and solving them proactively," says Carl Henning, deputy director of Profibus Int'l (PI) North America. "We're also reducing those 3 a.m. calls by making it easier for less-skilled technicians to change out failed devices without having to call in other engineers as often, or break out PCs and do a lot of configuration. In fact, Profinet has a simple device-replacement feature that allows a new device to be put in and be automatically recognized and configured by its network."
Dan Schaffer, business development manager for networking and security at Phoenix Contact, adds, "People monitor their physical health by weighing themselves and getting periodic cholesterol checks, etc. Monitoring a network isn't much different, except you use tools like simple network management protocol (SNMP) for particular ‘push' events on the network, free Wireshark software for monitoring and baselining traffic, and our FL View software to help visualize your network and quickly pinpoint bandwidth or link problems.
"From the highest level to the most detailed, SNMP and Syslog can push information to the operator or to a security incident and event monitor (SIEM), while tools such as FL View can show which devices are connected to which switch ports, which connections are experiencing heavy traffic or errors, and so on. Wireshark and similar packet-capturing tools can give you a deep dive into your network traffic and show what's going on on a packet-by-packet basis." On the hardware side, Schaffer says preventive measures such as network segmentation and using managed switches to act as firewalls can help keep the network free from unwanted or unauthorized traffic.
Jim Toepper, product marketing manager for Moxa's Industrial Ethernet Infrastructure division, explains that monitoring network health in today's applications must be automated. "There are so many industrial Ethernet nodes going live all the time that there's simply not enough manpower to check them individually, so we've got to have software that monitors and manages them," Toepper says. "Luckily, we've gained the ability to integrate network monitoring into existing SCADA systems via OPC-UA communications that provide a bridge to many network components."
To help its network devices report on their status and any issues, Toepper says that Moxa's managed Ethernet switches come with its MXview monitoring software, which is based on SNMP, has Internet Group Management Protocol (ICMP) pings and is offered free for up to 20 nodes. MXview also can recognize devices from Rockwell Automation and Siemens Industry. Some new MXview 2.3 features added at the end of 2013 include the ability to show Power over Ethernet (PoE) links, display power levels on the line, monitor up to 2,000 nodes from one PC server and deliver more detailed reports. Toepper says that Moxa is adding several features to its MXstudio software this year, including network installation functions that scale to thousands of units, more monitoring capabilities, and troubleshooting software that gathers configuration files from switches, checks if there are any conflicts or corrupt data in their configuration files, and then sends a Zip file to alert Moxa and the user.
Strength in simplicity
While network health can be evaluated anytime during normal operations, it's especially important to examine a network when it's first installed or upgraded to new equipment and communications capabilities. If the revamp reduces the network's complexity, then it's more likely to be more healthy too.
For example, the Mercamadrid wholesale market covers 547 acres, and includes separate facilities for meat, fish, fruit and vegetables, that feed more than 12 million people in and around Madrid, Spain. The huge facility's Mercado Central de Carnes processes, packages and distributes close to 137,000 tons of beef, pork, lamb and other products per year, and its top 25 distributors share 10+ acres of cold-storage and handling space in a three-story building. The meat market's natural gas-powered cold production plant provides ethylene glycol and liquid ammonia as heat transfer media for each distributor's refrigerators via 43 miles of pipeline manifold, plus other fluids for cleaning and maintenance. The plant's 23 compressors consume almost 23,000 cubic feet of natural gas per hour and generate 5,590 kW per hour to maintain required temperature levels.
However, control for the plant and its 26 cold-storage units was based on almost 15-year-old, stand-alone PLCs that exchanged data and populated a SCADA system via an outdated Modbus Plus network with a low 2 Mbps throughput. This whole network suffered from obsolete equipment, scarce spare parts, limited interface support and incompatible hardware. It was also difficult to maintain 27 separate control programs, and their ladder logic format made it difficult to program communications or other functions. All of these issues made it impossible to upgrade the cold plant or implement new control programs to improve chiller performance.
As a result, Mercamadrid sought help from Madrid-based system integrator Optomation Systems, which recommended replacing the old controls network with new operator interfaces and an Ethernet TCP/IP network using dedicated Ethernet switches in a less-costly ring topology with a supporting power ring, and linking the cold plant, maintenance areas and all 26 client, cold-storage units (Figure 2).
Figure 2: The cold production plant at Mercamadrid's Mercado Central de Carnes in Madrid, Spain, revamped and simplified its network with Ethernet switches that find the best path between points and are arranged in a ring topology to manage and distribute refrigerant fluids to 26 cold-storage units.
Source: Opto 22
New controls at the cold plant include a programmable automation controller (PAC) from Opto 22, which connects to an existing PLC at each of the 23 plant compressors. Optomation adds that the PAC uses Modbus RTU to enable start/stop commands and setting of operational values according to an algorithm that constantly checks the refrigerant requirements of all client units.
The integrator says its controllers, network and users also benefit because it created one common software program for its controllers and HMIs on the 26 client units, which can be downloaded to each one and greatly simplifies refrigeration operations and network performance. It built the network's unified software program by using the PAC's dynamic subroutines, object pointers, file manipulation, XML parsing and other features. As a result, specific parameters for each client unit are defined by simple text files uploaded over the network by FTP into its PAC, including its I/O definition and configuration, number, type and name of chambers, number of chillers by chamber and so on. The common control program uses this information to adapt itself accordingly, similar to a batch production control program.
Finally, Optomation added its Web Portal software to the meat market's refrigeration network, which allows it to collect data from all the controllers, fill an SQL database, generate dynamic Web pages and report on the activities of the cold plant and client units to any authorized user with an Internet browser. Conversely, users can monitor their units and download new module configurations, while Optomation can check in and remotely troubleshoot new instrumentation tasks or issues without being onsite.
"The dam broke on Ethernet years ago, and everyone's been embracing it since then," says Ben Orchard, Opto 22's application gineer. "Now everyone is pushing IT-based data-to-cloud services, so they can have even more big data available everywhere, all the time. Of course, manufacturers want in on this too. Gone are the days when plants and applications could be down for an hour or even half an hour."
Inoculated by Ethernet
Not surprisingly, one of the best ways to improve network health is to make the transition to Ethernet because it's so pervasive and has so many IT-based diagnostics tools. For instance, Sandia National Laboratories (SNL) in Albuquerque, N.M., recently worked with system integrator VI Control Systems to replace the obsolete 386-based computers, controls and aging point-to-point networking on SNL's annular core research reactor (ACRR), and they opted for nine PCs running Windows and LabView software, I/O modules and timing hardware from National Instruments, and motion components from Delta Tau Data Systems.
"We chose Ethernet for communications because of its durability and prevalence in industrial and business networks," says Neal Pederson, VI's president. "A switching hub connects of the control system's nine PCs and six FieldPoint network modules. The network is isolated for improved system stability and security. Also, a DataSocket server runs on one PC to provide communications between the nine PCs. Each PC has write privileges to one DataSocket dataset, but all nine PCs can read from any of the data sets.
"The system handles time-critical operations by using discrete logic hardware or by using the NI timing boards. A program scan-time update and associated rotating graphics are displayed on every screen, which indicates proper operation of the LabView code. The primary ACRR computer control system works with a 50-msec loop cycle time. Three of the nine computers continuously monitor each other and use a watchdog shutdown function. As a result, if any one of the three computers doesn't update its watchdog through DataSocket every second, then the two other computers initiate a watchdog shutdown and terminate reactor operations. This protects against any computer crashes and network problems."
