Avoiding security vulnerabilities when connecting to the IIoT

Marty Edwards, director of Industrial Control Systems Cyber Emergency Response Team (ICS-CERT) at the U.S. Department of Homeland Security, spoke exclusively with Control Design about several issues that need to be addressed when deciding to open your network and share data.

Source: U.S. Department of Homeland Security

Fundamental security issues can be introduced when connecting control system environments to other environments such as business networks. Marty Edwards, director of Industrial Control Systems Cyber Emergency Response Team (ICS-CERT) at the U.S. Department of Homeland Security, spoke exclusively with Control Design about several issues that need to be addressed when deciding to open your network and share data.

ICS-CERT works to reduce industrial control system risks within and across all critical infrastructure sectors by coordinating efforts among federal, state, local and tribal governments, as well as industrial control system owners, operators and vendors.

Edwards brings more than 20 years of experience and a strong control systems industry focus to DHS. Before coming to the ICS-CERT, Edwards was a program manager at Idaho National Laboratory. He has also held a wide variety of roles in the instrumentation and automation fields, including field service, instrument engineering, control systems engineering and project management. Edwards holds a diploma of technology in process control and industrial automation (magna cum laude) from the British Columbia Institute of Technology.

CD: At Control Design, we appreciate the work you’re doing every day. Thank you for the letter regarding our “Tear Down This Wall” cover story. We’re definitely serious about cybersecurity, but perhaps, like many of the machine builders out there, we don’t know as much as we should. You said in your letter that incidents are happening daily. What’s a typical cybersecurity incident related to industrial control systems?

Edwards: They can be detected through a variety of means, and they can actually span a fairly wide range of incident types. Incidents range from what I call commodity-type malware which could be a Trojan design dealing with banking information that is proliferating around the Internet accidentally getting into an industrial control system and infecting the machines. Or it could range all the way out to a significant, advanced and persistent threat from a nation-state-level actor who is very surgically and specifically targeting that control system for whatever the reason is.

CD: Are they causing any type of damage, or what is the typical result when an intruder compromises the control system environment?

Edwards: We don’t have a lot of cases documented globally where actual damage has occurred. Probably the most widespread incident that’s been talked about is Stuxnet, which actually caused damage to process equipment. There’s also an incident widely recorded in the open press in Germany of a steel mill that may have had some type of malware impact its steel-making process. But, for the most part, the incidents of the malware don’t really cause much harm. It would be more of an annoyance if it were in the sort of commodity malware family. But certainly there could be loss of production involved if you have to take a system off-line to clean it up or if the malware somehow affects the processing or uptime of the control system itself causing it to go off-line, resulting in production loss.

CD: So, the Industrial Internet of Things is coming, whether anyone likes it or not. Some are saying it’s not secure, but nothing’s going to stop it. For the average machine builder though there are likely some safety issues they should be aware of that perhaps they don’t pay as much attention to. Are there things the machine builder and controls designer should be doing to address security issues?

Edwards: Yes, absolutely, and I have a lot of empathy for the control system designers because, before coming into this role in security, I was a control systems engineer working both for vendors and users. My background is in the distributed control system area that did continuous plants, but I certainly have empathy. I think the advice is to very clearly understand what your system or machine is designed to do, and it’s, for example, a life-safety type of application, or you’re doing some type of engineered controls where you have to prevent entry into an area; you want to make sure that those types of systems are completely 100% air-gapped or isolated from any corporate environment, any engineering type environment, any other networks.

The life-safety types of systems should be completely stand-alone and very rigorously protected from a change control point of view. As you get into other systems that don’t have a direct life-safety type of application, such as a process skid in a typical manufacturing plant, there’s a lot of impetus to connect those to your corporate environment and to your other control-system environments, and what you have to do is look at the risk of compromise or malfunction of that device versus connectivity from a business perspective. Usually it’s not the integrator that can make this decision; it’s the asset owner or the owner of the manufacturing plant.

The control system designers have to weigh the advantages of the connectivity with the disadvantages of the security risk that connectivity brings into the system, and then you have to protect it at an appropriate level. I think all too often these systems are shipped with an Ethernet card in the PLC backplane or Ethernet connection right on the processor, and people see that, and they just simply connect it to the corporate network and leave it with the default usernames and passwords. It's wide open, and the default security is often turned off. My wish is for the manufacturers and integrators to just take that first step, the first look, at how the security of this device is being configured for whatever application it’s being used.

Also read: U.S. government resources for cybersecurity

CD: That’s a great suggestion: Get started with cybersecurity, and don't just leave access wide open. There are a lot of wireless connections being made on the plant floor to a smartphone or tablet, for example. Aren’t those the same concerns as connecting to a business network?

Edwards: They’re certainly very similar concerns. I think that people tend to actually think about the security of wireless implementations a little bit more carefully before they actually roll them out because just the term “wireless” makes them think about security. So it’s not unusual for us to see wired installations that have absolutely no security and wireless that have at least some.

Of course, just by involving a wireless signal, now you have to start to think about how far that wireless signal propagates. Does it leave my property? If you’re in a manufacturing plant or a small facility, can somebody from outside the chain-link fence either inadvertently or intentionally access the wireless signal, and what security protection do you have in place there?

When it comes to tablets, smartphones and bring-your-own device (BYOD), I would urge companies to always, if they need to use a wireless device for human-machine interface (HMI) or process interface, only do so with corporate-owned devices that are under the control of the corporate security policy. I think that there’s a trend in the business IT world to let people bring their own devices to connect to the network, but, for these critical process control applications, it’s imperative that the devices be under the security policy of the corporate security folks.

And then you can also use enclaves, or, as we call them, demilitarized zones, to bring all your wireless devices in and then group them together before giving them access to any of the sensitive process control networks. Some pretty rigorous controls should be in place so users have to actually authenticate to the system. They need to prove who they are, with a token or some type of two-factor or multifactor authentication, before they’re actually allowed to make changes in a machine or process control environment.

CD: You don’t think that is likely with the bring-your-own-device type of scenario?

Edwards: It’s just a lot harder to enforce in the bring-your-own device scenario. You don’t know what the user has installed on the device already. There may be malware on the device that could compromise your process control environment. The IT world is building the policies, and in general when you look at the process control world we tend to lag behind the IT world by about 10 years, so I think it’s a very risky area.

We see a lot of issues in those areas, as well as in remote access. Employees have remote access from their computer systems at home, or vendors are provided remote access into their products for warranty or monitoring purposes. Those implementations need to be very carefully scrutinized from a security perspective.

CD: Are there cybersecurity concerns, not only from a wired perspective, but from a smartphone or wireless perspective, also?

Edwards: I think we see the gambit; we see both. In the wired implementation, we do see a lot of devices that we believe shouldn’t be available or accessible from the Internet, and yet they are. A person takes the programmable logic controller and inadvertently or intentionally plugs it into the corporate network, giving it an IT address, not realizing that that action could in fact expose it through whatever the corporate perimeter protections are to the Internet. This allows network search tools to map all the devices that are available. Then, in cases where there are default usernames and passwords, the level of effort isn’t very high for an adversary to get in.

CD: It is really no different than plugging in to the front of the controller at that point, if they leave it that open.

Edwards: Exactly, and sometimes we found that these are intentional, in a remote facility, for example, where there is a manufacturing plant and maybe a mile up the road there is an unattended intake for that plant, and it is off your main campus or property. In some cases we have seen implementations where people go down to the local electronics store and buy a DSL router, plug it in and get a phone line pulled in from the phone company; the device is banging off the Internet. Then they just go and check the bytes.

CD: And then a hacker, with a little bit of technical knowledge, can go in there and wreak havoc if they want.

Edwards: Absolutely, so you just have to take a really good look at what you are using the device for and what implications it has if the device malfunctions or incorrectly performs the command set that’s inside of it. If the control algorithms get rewritten or overwritten, what implications does it have to the process? Consider personnel safety, machine reliability, equipment damage or rejected product, and then put the proper security envelope around that.

CD: That's great advice. I have seen a hydroelectric application in a tropical location that had a wide open network. They connected a main control room and nine remote sites via phone lines and Ethernet. It was wide open. With little work, an adversary could shut power off to a good percentage of the population. It is a concern.

Edwards: The cost of protecting those types of installations has calmed down dramatically. You can buy relatively inexpensive end-to-end VPN solutions where you could encrypt all of the communications between the various facilities and take away a lot of the network exposure. It really doesn’t come up when the integrator is laying it out.

The integrator often doesn't go in with a security mindset. In an IT installation, if you are a business that had a corporate headquarters and you had six field offices in different states, no networking or communications engineer would think of running those types of communications open over the Internet anymore. You would never run your email or financial billing across open networks, so why do we think that’s OK in the machine and process control world?

CD: So that leads us to a lot of the device and control hardware suppliers who see the value of production, machine and device data. They're kind of the ones leading the charge for the Industrial Internet of Things. At NIWeek this past August, IBM’s Greg Gorman said that cybersecurity is just an engineering problem waiting to be solved. If that’s true, why not make finding the solution a top priority from a hardware standpoint?

Edwards: This is one of the big challenges for not only the Industrial-Internet-of-Things community, but for other communities and sectors such as building automation. I was at a large building automation conference several months ago, and, while walking the floor, I could see air conditioners and all kinds of heating, ventilating, and air-conditioning devices with little antennas. They’re all sending their data up to some post system. I saw what I think somebody referred to as the lick-and-stick sensor. It's a temperature transmitter that you peel off of a small cardboard card and stick it on the side of a pipe. It’s wireless. For power, it’ll harvest thermal energy from the pipe that it’s attached to, and it’ll transmit a signal. When you start looking at the computing power that it takes to implement basic encryption inside of these devices, you get into a very cost-prohibitive type of function. I was talking to one manufacturer, who I won’t name, that manufacturers thermostats for commercial and residential use. The manufacturer said their price point was such that they couldn’t afford to put in $0.50 for the encryption technology because it would price them out of the competitive market.

Also read: 5 continuous elements for effective cybersecurity

So as we get into this smaller, cheaper, more commodity-based sensor market, the industry is really going to have a hard time adopting these edge sensors in a secure fashion. The end devices do not yet have the horsepower needed for security and since they're a throwaway and disposable type device, nobody really fixes those kinds of things. You just throw it away and put a new one in. If that's the case, the security configuration will have to be almost default, out of the box, an always-on type of implementation to work. You really can't expect the end user to manipulate them in anyway.

CD: So plan for the minimal in many of the edge devices?

Edwards: Right. In those types of installations, my first advice would be to know if they're all wireless. If so, have them on their own private wireless network. If there is encryption or wireless security available, use it or the best possible security available on all of the devices. Then, prior to bringing those networks into your main machine or process control network, bring them in through some type of perimeter processing. Have walls with some very strict rules in place that control the inflow and egress of network traffic and continuously monitor those networks.

CD: How many or few are monitoring the networks?

Edwards: Another one of the big takeaways that we’ve seen during assessments of industrial facilities is that, although they may have their control system engineers that look after all the equipment, nobody's really monitoring any of the systems for security alerts or weaknesses. So, even if you do have some type of intrusion-detection system that's in place, that's monitoring the perimeter of your control network, typically it's only ancillary duties that the engineer looks at that console to see. "Oh, what was this? We had five login attempts last night from somewhere that we don't recognize."

We need to get it ingrained into the operational doctrine that monitoring the security of these devices and networks has just as much importance as keeping the networks running themselves.

CD: It’s certainly good to collect the security data, but you have to look at the data once it’s collected and make some decisions on it.

Edwards: You have to take the first step, which is to collect the data. All too often in this area we find that people have turned off login capabilities in the security area, even on the human-machine interface. For example, if you want to login with a username and password, a lot of the devices are not logging who logged in and at what time, so you can’t extract that information. The next part after you've collected the data is to assign real human capital and personnel to analyze the data on some sort of regular routine and basis. Look for anomalies; if you don't have somebody looking at it, it’s of limited value.

CD: So, the security protection can be questionable if you don't do the right thing and be proactive. I’ve recently seen a controller with Bedrock Automation’s cybersecurity layered and embedded in the controller, connections and interconnections. Do you see that happening in industry that can afford to pay more to have cybersecurity technology built-in?

Edwards: I'm not familiar with that specific controller, but I am aware that the vendor community has been discussing this for some years. The message back up to the security folks is that the end users are not demanding this. It isn't something that they're willing to pay for right out of the box. Hopefully that is shifting and people are willing to pay a premium for a product that’s secure right out of the box. I think it's inevitable that the community moves that way. I'm just somewhat disappointed that we haven't seen a lot faster change in that area.

It’s difficult because you have interoperability issues with legacy devices. If I come out with a PLC or some sort of control device that has great whiz-bang encryption right at the controller level, how does it interact or operate with the rest of my legacy equipment that's 20-plus years old and doesn't have the same features. It’s a complex problem that takes a systematic approach over several years. You actually have to think about how you're going to implement it, what it changes you're going to make to your overall system and how you're going to phase it in and then maintain it over the lifecycle of that asset.

CD: Do you see any new or current past or future Ethernet protocols, industrial protocols, for that matter, having a greater impact on cybersecurity, good or bad perhaps?

Edwards: Not really. I can't say that I believe that this challenge is protocol-specific or is of more concern with one protocol over another. We've seen issues and vulnerabilities in serial communications; we've seen issues and vulnerabilities with IP Ethernet type of implementations and even proprietary protocols over proprietary networks. Security has to be looked at from a system-of-systems approach. You have to look holistically across your entire installation and design security in from the ground up, whether that means putting the appropriate defensive layers in place around the less-than-secure devices or it means lobbying your vendors to provide you with hardened devices in certain high-risk areas of your process. It's important to group all of those things together.