Administering TCP/IP In Automation or Measurement Networks - Comments

[Svein]

Svein submitted a new PF Insights post

Administering TCP/IP In Automation or Measurement Networks

Continue reading the Original PF Insights Post.

 

[BvU]

Two things: One: figure 1 didn't make it.

Two: I seem to remember that Ethernet wasn't qualified for real-time solutions because of this collision detection / backing off mechanism (CSMA/CD): you can't guarantee a time limit for getting the message across. So a lot of manufacturers developed buses that had some kind of interrupt level facility. Is that still an issue ?

 

[jedishrfu]

Nice insight article!

 

[Svein]

Two: I seem to remember that Ethernet wasn't qualified for real-time solutions because of this collision detection / backing off mechanism (CSMA/CD): you can't guarantee a time limit for getting the message across. So a lot of manufacturers developed buses that had some kind of interrupt level facility. Is that still an issue ?

No, it is not an issue today. Switched Ethernet does not use CSMA/CD. Instead, the switches receive the packets and put them in the queue for the destination node. The only way you can lose packets is to have a massive overload of the network (meaning that the link to the destination node is busy all the time and the output queue is full).

In addition, IEEE 802.1D defines a mechanism for packet priority and virtual private networks. Packet priority is especially relevant for automation and measurement networks.

 

[JakeBrodskyPE]

I've seen numerous attempts to set up Plug-and-Work applications for industrial control systems. Damned few of them have ever served me well. There are too many ways that it can fail or do unexpected things.

I do not use DNS for naming ANYTHING in a machine to machine network.

1. It is a central point of failure. The rest of the network can be seriously bollixed up from the latency of looking something up on DNS --especially if the latency of the DNS goes up for some reason. This is a real time network and milliseconds matter. I hard-code IP addresses.

2. The value of DNS is that if the addresses change, the DNS can update the rest of the world. Well, I don't want the rest of the world poking in this real time network. Real time networks tend to be very static. They don't change for YEARS.

I am very cautious about ARP. Many routers and switches have ridiculous ARP timeouts of HOURS. If I swap out a broken controller, I don't want to wait more than a couple minutes for the ARP protocol to forward the addresses across the network. Many embedded devices do it badly too. I am not above setting static ARP caches for an industrial control system.

Simple Network Management Protocol isn't. It looks like a bunch of Computer Science professors sat around a table, ignoring everything that came before in SCADA systems design and then took some really cool ideas and implemented them very badly. The object tree was a great start. But ASN.1 was a disaster. It had no concept of an event, either. SNMP could have been much nicer that it is.

I don't use DHCP for automation either. DHCP takes too much time to address equipment after a power failure. Furthermore, the equipment is supposed to stay up for weeks and months at a time. Again, these are very very regular operations on very unchanging static network.s The advantage of DHCP is that anyone can join the network with minimal effort. Well, I don't want just ANYONE to join my real time networks. I keep my subnets small, and I control my address spaces with a tight fist. Everything is alarmed. Light up the wrong port and there will be a big yellow banner across every operator station that somebody is working on the network that needs to be watched.

Frankly, the Internet Protocol suite is a poor fit for industrial control applications. I use it because the software has been battle tested, because people think they understand it, and it gives them a warm fuzzy feeling. Unfortunately, the reality is that very few understand it well enough to diagnose anything. I've seen it fail in ways that make most IT networking guys wonder what hit them. I don't like complexity like this. I prefer to keep things as simple as possible. Use as few of the RFC suite protocols as you need to get the job done. Watch your latency, and beware of dependencies and single points of failure.

I have much more to say about this, and in fact, I've written chapters and edited a book on the subject. The advice that Svein offers here is well intentioned, but unintentionally toxic in some situations.

Jacob Brodsky, PE

 

[Svein]

I do not use DNS for naming ANYTHING in a machine to machine network.

1. It is a central point of failure. The rest of the network can be seriously bollixed up from the latency of looking something up on DNS --especially if the latency of the DNS goes up for some reason. This is a real time network and milliseconds matter. I hard-code IP addresses.

2. The value of DNS is that if the addresses change, the DNS can update the rest of the world. Well, I don't want the rest of the world poking in this real time network. Real time networks tend to be very static. They don't change for YEARS.

This is, of course, an option. The trouble doesn't surface until one device breaks and you have to replace it. What IP address should the new device have (you have carefully documented every device in your network, of course) and how do you connect it to the automation/measurement controller program?

I am very cautious about ARP. Many routers and switches have ridiculous ARP timeouts of HOURS. If I swap out a broken controller, I don't want to wait more than a couple minutes for the ARP protocol to forward the addresses across the network. Many embedded devices do it badly too. I am not above setting static ARP caches for an industrial control system.

Again, if you have to replace a broken device, it will have a new Ethernet address (if you are not going the "locally administered Ethernet Address" way). So you have a new device, you hardcode the same IP address - but without ARP, the device will never answer.

The advantage of DHCP is that anyone can join the network with minimal effort. Well, I don't want just ANYONE to join my real time networks.

That is not a problem, You can easily insert Access Control Lists in a DHCP server, ensuring that only your approved devices get a valid IP address.

Simple Network Management Protocol isn't.

Agree. But there has been several revisions to SNMP with updates and corrections (I think the latest is rfc 3411). It is steadily getting better.
You might be interested in the IEC 61850 standard (https://en.wikipedia.org/wiki/IEC_61850). This is a standard for managing large measurement and automation networks.

 

[JakeBrodskyPE]

All I'm going to say Svein, is go ahead and try living with this stuff. I have spent most of my career living with my creations through the entire life-cycle. The other Operators, Technicians, and Engineers know my phone numbers. If something breaks and they think I had something to do with it, I'll hear about it. I go on 24 hour call for one week every month and a half.

No, I don't hard-wire the ARP tables except in extreme cases. But I have done it to get around poor behavior in embedded devices.

Let me tell you a little story from over 25 years ago. I recall, as a young technician going to school at night, a consulting programmer who visited us to help commission our SCADA system (in the mid 1980s). He told us of the UCA effort and how it was going to unify the SCADA world. Everything would have objects and those objects would be standard, and it would all just drag and drop into a new nirvana of design and ease of maintenance. Well, we waited, breathlessly at first, and then curiously, and then we sort of forgot about it. Yes, they've been at it for a long time.

The IEC 61850 object model and the MMS/GOOSE transport has a major problem to overcome: The object definitions should match the devices in the field, NOT THE OTHER WAY AROUND! Yet, that's exactly what 61850 tries to do. And that's why, decades later, they're only just starting to see some meager traction. It works if you have a green-field, clean sheet design. But that's not what most users have. And if you try to mix GE, ABB, or Siemens gear, you'll be in for a rather unpleasant surprise. It's not as plug-and-play as it might first seem. I wish I could say something nicer about it because I would really like to have seen this work. But after decades of effort, they still don't have a lot to show for it --and I'm pretty sure that it's because they have firmly planted the cart in front of the horse. Designing hardware to meet a software goal is a non-starter in most engineering houses. Go try it some time and let me know how well it works.

If you'd like to see a sample of the problems the 61850 standard has faced, read the verbiage on the compliance certificate: "This device has not been shown to be incompatible with the standard." Doesn't that inspire confidence?

By the way, this same stumbling block is one of the reasons the FieldBus standard got stalled in committee for about 15 years.

Frankly, I'm tired of kids in cubicles who think a pump is an object. I'll slap a hard-hat on their heads, some steel toed boots on their feet, and drag them out to a pumping station and show them the subtle differences of each pump in the station. No, they're not easily summarized by objects. These things grow and morph, and change as the years go by. Unless you install and commission them all at the same time, these objects do not copy directly. They cannot be standard. A pump of brand X from 10 years ago is not the same as pump Y from two years ago, or pump Z from 50 years ago. That's the fallacy of trying to build standard objects.

Yes, I've used object models to handle identical devices in new installations. It saves some commissioning time there. But that's a far cry from the one true object model of everything.

I prefer simplicity. Keep It Simple, Stupid!

When you have to live with your creations in a 24/7 environment year after year, you too will understand.

 

[Svein]

I understand your frustration, it matches the stories I have heard from others (I was involved in ABB R&D for 15 years). And, believe me, the fieldbus guys were even more frustrated. Whichever fieldbus they were trying to standardize on, the customer had a different flavor. And if by chance the customer had a fieldbus the installation guys knew about, the system documentation were always missing/incomplete/out of date.

What we did way back when was to tell those people that they could standardize on Ethernet hardware - inexpensive and readily available. And the very cheapest Ethernet (the 100Mbit/s version) was way faster than the fastest fieldbus.

Ethernet as a hardware concept caught on, but immediately every vendor wanted to run a proprietary protocol on top of the Ethernet MAC layer - and as far as I know they are still at it. TCP/IP was nobody's choice (it was never meant to be an automation protocol), but it was the only thing that worked. The trouble was that the fieldbus guys still wanted to configure everything by hand, and could not understand that an Ethernet controller came with a built-in hardware address.

My purpose in writing this small guide was to tell those fieldbus guys: Hey, if you are running an automation protocol on TCP/IP, there are several tools available that can make life a bit easier for you!

The other Operators, Technicians, and Engineers know my phone numbers. If something breaks and they think I had something to do with it, I'll hear about it. I go on 24 hour call for one week every month and a half.

Oops - what happens if you have a heart attack and end up in hospital? Can others figure out what to do in case of a breakdown or will the whole thing stop because you are the only one who knows the system (and you are not available).

 

[JakeBrodskyPE]

what happens if you have a heart attack and end up in hospital? Can others figure out what to do in case of a breakdown or will the whole thing stop because you are the only one who knows the system (and you are not available).

No, things don't go to a screeching halt. We don't design our control systems that way. In any case, we have a staff of 10 control systems engineers. We cover for each other and we cross over various disciplines and projects. Obviously, if those on call have questions, they'll reach out to the person who touched a system last. If that person is not available, others will step in, but recovery will probably take longer.

There are no critical staff in our group.

 

[Greg Bernhardt]

Interesting Insight!