15 considerations for deterministic response with industrial Ethernet
A Control Design reader writes: Our factory manufactures subcontracted parts for an aerospace company, and we’re having a couple new production lines installed with quite a few smart devices. The system integrator insists that our network capabilities be upgraded to ensure real-time deterministic response. Is that necessary? What are the benefits? Do the existing wireless components require special considerations? Is deterministic response part of industrial Ethernet?
Also read: Why time-sensitive networking will change everything
Answers
Weight determines space worthiness
Yes, determinism is possible with industrial Ethernet. But one must be cautious.
Aerospace manufacturing is all about weight. It is extraordinarily expensive to put mass into orbit. And it goes without saying that a vendor does not want to be the reason for a failed mission. How can a manufacturer minimize weight for every part without sacrificing quality or safety factor?
It is Control Theory 101 that in the manufacturing process the closer one can align outputs with inputs—reaction time—the tighter the tolerances that are possible. In other words, the faster the communication scheme, the less safety factor needed. This ensures manufactured components will have minimum weight and a verifiable space-worthiness.
Wireless communications are an issue. Wireless is half duplex—separate update of input and output information. Wireless also involves significant time jitter—different update time periods per cycle. This means the user must add code to combat limitations and must overshoot the process to make sure that the component being manufactured meets necessary quality standards. The result is more weight per part and additional development effort to attempt a difficult battle. Costs go up when malformed manufactured components are tested and determined to be not space-worthy and must be discarded. In aerospace, raw materials are expensive. The more we can eliminate control issues, the less raw material is needed, which will lead to better success in space-worthiness testing.
A real-time, deterministic response can be achieved with industrial Ethernet, but it is difficult, if not impossible, with normal switch-based, IP-based communication. The control processor is never the weak link; the weak link is always the ability to update the input/output devices quickly and deterministically. And there is a solution. The EtherCAT industrial Ethernet system uses standard unmodified Ethernet in a special way. Frames are Ethernet-based, but not IP-based. The EtherCAT protocol uses one frame to update all input and output devices each cycle. The result is a much more time-determinate method of updating inputs and outputs.
In addition to updating the control system in one cycle, EtherCAT makes use of the concept of distributed clocks to further improve synchronization. All devices can share a system clock and update their process images at the same exact time. This makes motion control more precise, and less effort is needed to achieve minimum tolerance. This does not require additional cost; the synchronization mechanism is built into every EtherCAT device from any vendor and only needs to be enabled.
EtherCAT is not a niche protocol. It is a popular communication standard that has been in existence for almost two decades. EtherCAT is an open and globally accepted protocol with more than 6,500 members in the EtherCAT Technology Group. In summary, EtherCAT is being utilized more in the aerospace industry because of the low latency and deterministic synchronization.
Robert Trask / North American representative / EtherCAT Technology Group / www.ethercat.org
Interoperable solutions
Ensuring a real-time, deterministic response is a must for next-level smart manufacturing and the Industrial Internet of Things (IIoT). While standard Ethernet cannot support this goal, per se, different solutions have been developed to build suitable industrial communication networks.
While the limitations of standard Ethernet haven’t been an issue in the past, they can affect future factory operations, which demand increasingly powerful communications that can support real-time, time-critical transmission of control data. Different solutions have been developed to make this network technology more deterministic, and thus futureproof.
Ethernet systems are being created to ensure deterministic communications by offering 1-Gb bandwidth. The different solutions to achieve determinism often consisted of custom, proprietary industrial Ethernet protocols aimed at addressing specific tasks or domains. Thus, their scalability was limited, as each was tailored for a specific application area. Furthermore, while many of them could be considered open, they would not be compatible with each other.
To address these issues, some of the major developers of industrial Ethernet protocols have partnered to ensure interoperability between their solutions. For example, the CC-Link Partner Association (CLPA) and Profibus & Profinet International (PI) created a joint specification to allow for easy interoperability between CC-Link IE and Profinet.
Efforts have been made to build real-time determinism in standard Ethernet. More precisely, a set of IEEE 802.1 standards operating at Layer 2 of the data-link layer has been developed. The result is time-sensitive networking (TSN), which helps industrial Ethernet to deliver deterministic communications and ensure a reliable delivery of data between endpoints.
TSN technology has huge potential, providing a robust and reliable infrastructure and supporting control-based applications from the embedded world to the cloud, and it is being developed rapidly to accommodate for higher-level functions, as well as network safety and security. It can already provide substantial benefits to manufacturers, supporting the creation of highly interconnected plants. Therefore, businesses that want to prepare their factories for the future of manufacturing and implement IIoT strategies should act now, combining TSN with high-performance Ethernet.
Daniel Weiss / senior product manager / Newark / www.newark.com
Synchronized control network
If you have a lot of servo motors or similar machines that need to be synchronized, then your network must have real-time deterministic capabilities (Figure 1). The need to update your network depends on your current setup. If you’re already using an Ethernet network, you should be able to use the same network for the new production lines, although you’ll need to check that the network can handle the increased traffic. If you’re using a fieldbus network, it’s likely that you’ll need to upgrade to an Ethernet network. There’s been an increasing trend in the industry to replace fieldbus networks with Ethernet networks, so now might be an opportune moment to make this transition. The protocols are very similar, so the change is less scary than it might seem.
Upgrading to an Ethernet network offers two main benefits. It improves performance, meaning that equipment might be able to run at increased speed, which in turn could increase efficiency. Your network will be also able to process more information, providing opportunities for better diagnostics and additional IoT data, making it possible to identify problems or plan maintenance updates before production is impacted.
If we assume that no control—deterministic—data is transferred wirelessly, then no special considerations are required; the diagnostic and IoT data can be transferred on the existing wireless components. If control data is transferred wirelessly, then you need to ensure that the wireless components are sufficiently robust and that all dedicated wireless components are using the correct wireless technology such as Wi-Fi, Bluetooth or cellular for the relevant application.
Yes, all industrial Ethernet protocols are required to provide deterministic real-time responses. How quickly the real-time responses are sent depends on the industrial Ethernet protocol and the target applications. The messages can be sent in seconds, milliseconds or microseconds.
Joakim Wiberg / head of technology / BU Anybus / www.anybus.com
Calculating system latency
With adding smart devices to the network, the amount of data that needs to be communicated over the network will increase exponentially, so the demand of the existing communication network will be much greater. Ethernet data rate is much higher than a lot of traditional fieldbus systems, such as Modbus RTU or CANopen. The data rate for the devices that support industrial Ethernet is 10 Mbps/100 Mbps or higher, depending on the type of the smart device. In the industrial factory automation system, the industrial Ethernet network needs to be accurate and must be deterministic. Long latency in the network means that you will build fewer products, reducing throughput and ultimately impacting revenue. In other words, the better the latency the lower cost per unit of production.
Ethernet physical layer device latency occurs when a frame is being transmitted or received. With adding smart devices to the network, the transmitted time and/or received time can be increased. For calculating the system latency, you must consider all different components and cabling. So, if you can minimize the latency and its variation from cycle to cycle, you have the ability to add more nodes on the network bus, helping you to get the most from your system. Time-sensitive networking is an upcoming new technology that aids in redundancy, enhances traffic control and timestamps each message in an Ethernet network. It allows for flexibility, scalability and support for small devices, as well as big data in time-critical communication. To date, it is optimized for wired networks, as opposed to wireless networks.
Overall, to answer this question you must calculate the latency of the entire network and find the optimal point between the cost of improving the latency and the cost of network upgrade. With wireless networks there is a finite bandwidth that is available, and, if it is used up by existing devices on the IT side, the OT functionality will overload the system and result in lost information and errors in communication. There are ways to isolate the IT side from the OT side via routers or even changing frequencies or channel bands, but ultimately it does involve the IT/OT side converging to plan for future upgrades and maximize results.
Sanaz Kanani / project manager, network connectivity and IIoT / Weidmuller / www.weidmuller.com
The right level of determinism
When installing new automation control technology, upgrading to the latest proven network specifications is a best practice, when practical from timeline and cost perspectives. This will ensure that the investment will last as long as needed and be able to integrate with future innovations. The practical meaning of real-time determinism depends upon the intended application. Industrial automation Ethernet protocols can meet the speed requirements of the overwhelming majority of process automation applications. However, in many more discrete and hybrid applications, it is necessary to adapt standard Ethernet communication networks to ensure ultra-fast delivery, response times and device synchronization. This can be accomplished via a segmented network design with Layer 2 network switches, quality of service configuration for switches and standards such as IEEE 1588 or the new IEEE 60802 TSN Profile for Industrial Automation.
The level of determinism and the appropriate mitigating design and/or communication network solution depends on the business needs of the automation installation. The benefit of designing higher levels of determinism into a network is that the addition of new devices and data onto the network in the future is less likely to cause undesirable issues such as unacceptable jitter or lost packets. It is possible to use a wireless physical layer for a basic deterministic application with appropriate design parameters such as additional repeaters, a clear line of sight between wireless nodes, minimized external interference and/or safety network protocols that can mitigate the impact of temporary loss of communication.
Determinism is a significant part of industrial Ethernet and was one of the key driving forces behind the creation of communication networks intended specifically for automation. The key is to assess the lifecycle constraints and needs of the application and to use that to determine what level of determinism is necessary and to then choose the right solution based on a cost/benefit analysis.
Dr. Al Beydoun / president and executive director / ODVA / www.odva.org
TSN doesn’t have to be an all-or-nothing approach
Your integrator likely has some very good reasons for the proposed upgrades. Depending on which applications are sharing the network, real-time deterministic networking capabilities, such as those defined under the IEEE 801.2 time-sensitive-networking standards, may be necessary to guarantee the safety and reliability of automated systems.
If you have multiple applications, some of which are time-critical, sharing the same network, TSN traffic-shaping capabilities can ensure that the time-sensitive applications get priority. In industrial automation applications that involve control, determinism is an absolute necessity. You need some level of determinism to ensure a control loop is closed in a specified interval of time.
There are many ways to achieve such determinism in your network, such as:
- overprovisioning the network
- fair-weighted queuing
- providing scheduled time slots for each application to transmit on the network
- preempting lower-priority traffic
- providing scheduled time slots for certain traffic classes
- synchronizing network access.
The right approach for your network depends on several different factors: control loop cycle time, packet size, the presence of potentially interfering non-control traffic in the networks or the number of network hops. In general, existing Ethernet-based fieldbuses provide for one or more of these mechanisms. TSN standards provide all of them.
Building in TSN capabilities helps you now, but it’s also a future-proofing issue. IT/OT convergence is maturing, and the demand for Layer-2 IT support of production line technology and centralized resource management will continue to rise in the next few years. Now is exactly the right time to begin thinking about adding deterministic capabilities to an industrial network; and you don’t have to take an all-or-nothing approach.
Deterministic capabilities do not have to extend to every segment of the network. In general, wireless networks are less deterministic than wired. If your network includes existing wireless components, it’s a safe bet that their use cases are a bit more tolerant of packet loss and longer cycle times than the most time-sensitive portion of your application. As you add new wireless devices and access points, however, it is worth planning to eventually extend deterministic capabilities to wireless network segments. Already, wireless 5G technologies are providing great improvements in determinism and reliability; true TSN capabilities for both Wi-Fi and 5G networks are on the horizon.
Dave Cavalcanti / principal engineer at Intel and chair of the Avnu Alliance Wireless TSN Workgroup / Jordon Woods / director of the Deterministic Ethernet Technology Group at Analog Devices and chair of Avnu Alliance Silicon Validation and Certification Workgroup / www.avnu.org
Data when and where you need it
Without knowing what network capabilities need to be upgraded, in general, the answer is yes: an industrial Ethernet network needs to be deterministic. In a way, that’s what separates industrial Ethernet from regular Ethernet. I say, “in a way,” because each industrial Ethernet protocol does it a little differently. Industrial Ethernet isn’t just one thing: it’s a broad name for protocols employed to automate factory control.
Some industrial Ethernet protocols rely on solutions designed for the office/IT environment—for example, TCP/IP, UDP/IP—for real-time control. Other industrial Ethernet protocols close their network so that only their traffic can exist and ensure determinism that way. And others utilize Ethernet but eschew other features that might negatively affect determinism. Full disclosure: Profinet takes such an approach, and I’m from the organization responsible for the Profinet protocol.
The benefits of industrial Ethernet are pretty clear: deterministic control in order to ensure data arrives where it needs to be in a well-defined amount of time, and no later. Speed and determinism are critical in factory automation. For example, in the consumer world, if I open a web browser and sometimes a webpage loads in 200 ms and other times the same webpage loads in 400 ms, it isn’t a big deal. On the factory floor, however, if packets sometimes take longer to arrive than they are supposed to—an effect we call “jitter”—now all of a sudden the line shuts down: it is a big deal.
Each industrial Ethernet protocol approaches wireless differently. Some industrial Ethernet protocols employ it natively, while others do not. Generally speaking though, you can expect cycle times and response times will be longer over wireless than with wired Ethernet. But for many applications the cycle times achievable with wireless are perfectly acceptable for an application, and so it gets employed. Use cases include moving machinery, automated guided vehicles (AGVs), automated mobile robots (AMRs) and rotating machinery. Often 802.11 (Wi-Fi) or 802.15 (Bluetooth) will be utilized, although proprietary options are available, too, for special environments. Besides the special considerations regarding achievable cycle times, radio and spatial interference should also be considered. Industrial wireless access points are available from various manufacturers that have features built in to help mitigate the effects from the rough environment on a factory floor.
Again, each industrial Ethernet protocol handles determinism differently. Some industrial Ethernet protocols attempt to achieve determinism utilizing tools from the office or IT world, such as TCP/IP or UDP/IP, while other industrial Ethernet protocols achieve determinism by simply closing the network. Without speaking for other industrial Ethernet protocols, I can say that with Profinet, what we do is a bit unique: we leverage the richness of standard unmodified Ethernet whilst also putting mechanisms in place for real-time response. The Profinet industrial Ethernet protocol is open; any Ethernet-based protocol can coexist plainly on the network—HTTP or OPC UA. For automation control-related traffic, Profinet skips the TCP/IP and UDP/IP layers that otherwise hurt determinism, going straight from the Ethernet Layer 2 to the Application Layer 7 of the ISO/OSI Model.
Michael Bowne / executive director / PI North America / us.profinet.com
Timely and accurate control
In all control systems, it is important that the controller receive information about the system under control that is both timely and accurate. “Timely” means at a higher bandwidth and small enough delay to meet the system responses required. “Accurate” means more resolution than is required in the final part tolerance. As with all engineering problems there are trade-offs between performance, flexibility and cost. Deterministic, low-latency networks help to make this possible. Therefore, in the use case you describe, the answer to your question is “probably.” It comes down to what the smart devices are doing and how a non-deterministic response affects your production process. So, let me start with a list of assumptions. First, I will assume that the smart devices being integrated are indeed integral to your manufacturing process. Second, I will assume that inconsistencies in your production system’s timing is not acceptable to your process.
If both of those items are true, smart devices can indeed provide real benefits to your control system. For example, for the device itself to be smart, it must have its own processor. Many automation suppliers use smart devices to offload some of a system’s overall functionality to a local, distributed controller. Think of a smart light bulb with an integral controller. New smart light bulbs come with features such as integrated timers, hue control, ambient light sensors and several other integrated controls. Likewise, in automated systems, sensors, motors and actuators can become smart and therefore perform some of the tasks once reserved for a central control or even distributed PLC and motor-drive control system. Because these control systems are local and dedicated to the device’s functionality, that local process can be optimized with the local smart-device controller. When these smart devices are enabled with the ability to respond to other smart devices, completely decentralized control architectures can be envisioned. Another benefit to going this direction is that each unique smart device becomes easier to troubleshoot and maintain. In our light bulb example, instead of trying to figure out whether the timer, bulb or some centralized control logic is to blame for a failure, simply change the smart bulb, which contains the complete functionality of the system, and you are back up and running, making money manufacturing aerospace parts.
Back to the question about upgrading your wireless network to be deterministic. Do you need to do this? An assessment of the bandwidth is required of each localized control and the interactions they need to have with one another. If this is truly a decentralized process, then it may not be necessary to upgrade the network. I am assuming it is now a standard TCP/IP network, which is non-deterministic and load dependent. However, if the localized controllers require information about the other local controllers in real time, then upgrading the network is likely needed. Or if the overall control of the line requires central processing and coordination by a centralized controller connected to several smart devices you also likely need to upgrade the network. Therefore, the communication between these devices should be on the same deterministic controlled heartbeat. In old-fashioned point-to-point wired systems, the data you get at the single, centralized controller is assumed to represent the real world as it exists at each moment in time. However, even in those systems, the multiple controllers and motor drives have always required a deterministic communication network. With smart devices, they become controllers. Therefore, the wireless network they talk on also should be deterministic, such that each device gets an accurate picture of the world in which it exists.
Many industrial Ethernet networks are not deterministic. When determinism is required, an evaluation of the network technology should be done considering the determinism, timing delays, network topology and cost.
Patrick Wheeler / product manager / Aerotech / www.aerotech.com
Tight tolerance and control requirements
Robustness and deterministic real-time communication are two major factors differentiating industrial Ethernet from standard Ethernet. Industrial applications have a very tight tolerance and control requirements, which require the use of specialized protocols and devices supporting those protocols. The timing requirements can be less than 100 microseconds. The infrastructure required to interconnect the devices like sensors, PLCS and motors is not generic for industrial Ethernet and depends on the protocol implemented on the plant floor. This can range from Profinet and Ethernet/IP to EtherCAT, to name a few of the protocols used in industrial applications.
There are different deterministic levels that dictate the choice of the network device to be used. The total throughput and the number of devices also impact the network infrastructure so that it can support the bandwidth required currently while also leaving room for future expansion. A Gigabit network device may be more cost-effective in the long run.
Wireless in general requires special considerations during installation to minimize interference. Mesh vs. point-to-point topology, latency, distance and max bandwidth limit the range of applications that can be implemented over a wireless network.
Harpartap Parmar / senior product manager / Contemporary Controls / www.ccontrols.com
Ancillary support to PLC control
I don’t see edge or cloud functionality taking the place of core control systems, at least not yet. Edge and cloud solutions provide ancillary support to PLCs, executing computations and feeding data to production, quality and maintenance optimization systems for machines, lines and plants.
Many control-system tasks are evolving to run reliably and efficiently in PC-based devices, but safety, motion control and other critical systems requiring time-sensitive response and extreme uptime assurance are better-suited by PLC control. This preserves critical functionality in the event of an HMI, IPC or edge-computing operating system failure.
John DeTellem / TIA portal product manager / Siemens Industry / www.siemens.com
Hybrid control: multi-functional platforms
To leverage IT capabilities while maintaining PLC robustness, we are seeing more hybrid controllers, which run separate PC-based and industrial cores in parallel within a single hardware device. We refer to these as multi-functional platforms (MFPs). Some of these MFPs are built on PC-based systems, running the core operating system, PLC application and industrial edge algorithms in parallel, yet independently.
The key for successful industrial edge-computing deployment is quickly analyzing data to optimize real-time control, instead of sending data directly to the cloud for all processing. The integration of IPCs and PLCs is the future of edge computing.
Luis Narvaez / basic automation and industrial security product manager / Siemens Industry / www.siemens.com
Networked functional safety
The necessity for implementing a real-time/deterministic network depends on the specific application scenario. For example, in a smart farm, a few seconds of delay for water irrigation probably will not cause any problem. While in the scenario of autonomous driving, a few seconds of delay for steering control could result in a serious accident.
As can be seen, one benefit with a real-time/deterministic network is to maintain functional safety, especially in areas of autonomous driving or aerospace flight control. Another potential benefit with a real-time/deterministic network is to improve operational efficiency. Again, in the scenario of the smart farm, though the timely control of water irrigation/pesticide spray is not critical, it could possibly reduce the water/pesticide usage and thus avoid unnecessary pollution.
Naturally, the existing wireless component will need to be re-evaluated, with the emerging/popularity of 5G networks or other wireless technologies. If the communication data is time-sensitive and mission-critical, migration to a real-time/deterministic network will help your products/services to remain competitive and future-proof.
And, yes, regarding the new emerging fieldbus technology of the recent decade, almost all industrial Ethernet technologies embrace determinism and real-time control in their initial design and implementation.
Jerry Lian / product manager / Advantech / www.advantech.com
High-speed vs. low-speed devices
It really comes down to understanding when determinism is required in a machine-control application. Deterministic networks are fully synchronized and have very low jitter and packet loss. Determinism is important in high-speed control applications because the machine controller cannot adequately control with relatively old information while trying to control devices that aren’t responding to commands when they’re told. If the signal of a high-speed registration sensor on a printer is being received several milliseconds late and the servo is late in responding to commands, the print quality will be noticeably weaker. So, yes, determinism is required for high-speed control applications.
However, not all devices on a machine require deterministic networks. Devices like HMIs, database connections and lower-speed control devices, such as oven temperature sensors or indicator lights, are slow enough that they do not require determinism. These types of devices can continue running on the existing network but could eventually be limited by total bandwidth. Wireless connections traditionally have issues with latency and connection stability, so migrating toward a wireless network would take some significant consideration. For reliability purposes, we prefer to keep control devices hardwired over Ethernet. Where problems can occur is when real-time determinism is being run on the same network as traditional TCP/IP communications. Network protocols will either use time slicing or the precision time protocol to achieve near real-time determinism on the same network cable. This results in lower bandwidth for all of the devices on the network and limits the network cycle times that can be achieved.
For these reasons, we use the dedicated EtherCAT network for real-time deterministic communications and standard Ethernet for all other Ethernet related network communications. This separation allows the devices that need high-speed control to run with determinism at very low network cycle times, while allowing all other Ethernet devices to fully rely on the randomness of Ethernet. This allows existing network structures to still be suitable, even when upgrading control systems.
Certainly, as data grows, bandwidth limitations exist. But that’s also why we’re seeing the emergence of low-weight protocols like MQTT.
Paul Anderson / technical manager / Omron / www.omron.com
Real-time vs. deterministic
When talking about industrial networks, real-time and deterministic responses are independent factors. The time an application requires an event to happen is defined as real time. A high-speed motion control application could have a real-time value of 250 microseconds. On the other hand, real time in HVAC applications is on a scale of minutes. For example, if a building occupant turns up the thermostat and feels the temperature change five minutes later, the occupant’s expectations have been met and the HVAC system response time of five minutes is “real time.”
Deterministic response is also a time factor, like real time, but it is based on user requirements for inquiry response time. In the HVAC example, the deterministic response would require that the user receive a response within five minutes—every time, and with no exceptions. In short, both real-time and deterministic responses are application-dependent. And deciding whether to upgrade ultimately depends on your return on investment or if your application requires it.
Manufacturers use industrial networks to improve their productivity. If a network upgrade would increase productivity, then the investment is warranted. If the upgrade wouldn’t, then the network will become more complex, and the upgrade will likely be more detrimental than beneficial.
Phillip Marshall / CEO / Hilscher / www.hilscher.com
