In any industrial maintenance shop, you will find diagnostic tools such as handheld scopes, multimeters and electrical test equipment.
Increasingly however the crew is becoming responsible as the first line of defense for OT networks. Software tools are becoming more and more prevalent in the maintenance toolbox.
Wired Ethernet networks can be symptomatic in various ways. Because most are carrier-sense multiple access with collision detection(CSMA/CD), the devices connected are built to behave in a certain way. It is when these devices do not behave properly that we experience network issues, and sometimes it is like looking for a needle in a haystack.
We need tools to diagnose wired network issues and discover which device or devices may be causing the observed anomalies at an affordable price.
There are many options available to the IT guys since most diagnostic tools are targeted toward that group, but the OT crew may not know what’s available to them if they are responsible for the OT network.
SolarWinds has a very common toolset for IT peeps. While comprehensive, the tools are expensive and may take a bit more base knowledge than an OT person or department has.
I looked after a distribution center in Ontario, Canada of more than 1.3 million sq ft, and the maintenance network employed maintenance-owned switches and also borrowed ports on IT switches and routers.
We had a problem in one of the nine buildings where computer data was not being transferred on time to a PLC-5. I discovered Intravue, now owned by Panduit.
I installed the product and was amazed by how it automatically discovered most of the devices on my network. Now in real time, and in conversation with Mathew Gier, product manager for Intravue, the software uses multiple options to determine who is online within a given IP address range. My application was simple—75 devices all in the same Class C address. Gier mentioned that Intravue can support up to 2,048 nodes in a single install. Multiple installs can support more than that.
I would suggest that if an OT LAN was greater than 150 devices, we may be overwhelmed, but, with the Industrial Internet of Things (IIoT), that will be probable.
How I discovered my issue was by the ping diagram that Intravue creates. It largely uses SNMP and LLDP to discover switches, routers and devices that are attached to each based on the management-information-base (MIB) tables in the device. It was determined that the PLC and computer were on the same switch and pings to the computer were taking a very long time, relatively speaking.
The packet size was small, and the activity was limited so it pointed to a chattering Ethernet card in the computer. The card was changed with knowledge—nothing worse than changing something, and the problem goes away, but you don’t know why.
I updated my install of Intravue to discover that it had matured very nicely. The network is still self-discovered; it allows you to put icons in for the devices; and the ping graphs are very precise. I fired it up on my home network since I have been having an issue of certain ghost happenings. Periodically my virtual machine interface would restart.
Thinking it was a software issue I reinstalled the interface. It was only when Intravue was running that it indicated that I was losing pings from my switch, and all devices connected to the switch were being taken off-line. Need a new managed switch. Very cool.
There is a cool function that allows you to put a graphic of your plant into the mix and then place the devices where they are physically located. I would have loved that when I was looking after 1.3 million sq ft.
The data logging gives you an overview of the threshold violations for the system, as well as for each device. You can link device to files, such as PDF files of operating manuals, which could prove to be valuable in times of panic.
Part of the threshold data is a bandwidth violation which will identify problem children such as a chattering network interface card (NIC).
Intravue is real time, as well as historical. Looking in the rear view mirror, data logging is a must when trying to figure out what happened after the fact. It is real time, as well, but I have found the historical information invaluable.
This network tool works, and it is in the industrial sphere and available to be used by non-network people. It is intuitive and vendor-neutral. It can be installed on any computer that has a Web browser.
This means that an OT system can be supported by OT people. Not disrespecting the IT spectrum, but sometimes it’s best to look after our own backyard.
Inexpensive self-diagnosis maybe isn’t so bad.
