Recommend architecture/protocol to sync data between MCUs

[A.J.710]

TL;DR Summary

Have a sensor/controller board pushing large amounts of information to a communication board and need a more professional way to synchronize them.

This started out as a simple controller only but kept advancing without ability to change the existing architecture and now it's getting too messy. I'm seeking recommendations on architecture or C libraries or anything to help clean this up because a major change is needed and I'd like to seize the opportunity to revamp the architecture.

Without giving too much info on use case etc.:

Board 1 is an MCU with an array of sensors including analog, i2C, and UART that controls multiple PWM and GPIO output channels. Timing is critical and control loops are tight. This board is self-sustainable but can be synced with Board 2 over UART.

Board 2 is a communication board. It syncs with Board 1 over UART and functions as a user interface over an ethernet connection. In order for information to be available to a user instantly, this board has a cache that is basically a copy of all of the parameters coming from Board 1. Synchronization is handled in the background sometimes the info can lag slightly behind.

The main issue here is the syncing of data between these two boards. This was never intended to get so advanced but after various additions, we are approaching levels of now hundreds of parameters needing to be synced between these two boards and of course the desire to be instantly available and updated. The protocol for information transfer was developed by me from scratch but it is getting unsustainable at this rate of expansion. Basically a packet structure of several bytes including metric category, sensor or controller type, sensor or controller number, data, and a parity/check to verify receipt.

Board 2 is always running an open UART read loop so any updates Board 1 sensors have are pushed immediately and stored in Board 2 cache.
Board 1 similarly is always listening so Board 2 can send updates induced by the user ethernet interface. It's programmed to receive, update internally, and then respond with that same parameter's data to Board 2 so that the cache is updated to reflect the newest update or old data if for some reason it can't be changed or is locked out or something.
Since everything is basically running in loops, this is getting too much to handle. Especially during board initialization where Board 2 polls Board 1 for every single parameter to fill up its cache. This is a sequence of like 1,000+ UART transfers every time it turns on and this takes forever. Also with so many parameters, updates are starting to get missed because they can't get processed in time due to the such critical timing of Board 1's functionality. This is where the opportunity to revamp is because I have to double the capacity of board 1 and in the meantime can update the MCUs and firmware on both boards to accommodate. I don't have much experience with advanced communication architectures and stuff so that's where I am looking for recommendations. Any help would be greatly appreciated.

 

[Rive]

UART is a bottleneck there and while some tricks might help, I think fundamental changes would require a different, lot faster channel of communication.

 

[berkeman]

UART is a bottleneck there

It sounds like it. @A.J.710 -- What protocol is this UART using to communicate with the PC? What is the datarate? Have you considered high-speed USB instead?

 

[pbuk]

I have started using the RP2040 as my default MCU. The UART runs fine at 921,600 Mbs (I believe it supports up to 4 MBps): is that enough for your application?

 

[A.J.710]

It sounds like it. @A.J.710 -- What protocol is this UART using to communicate with the PC? What is the datarate? Have you considered high-speed USB instead?

The UART is used to communicate from the controller board to synchronize with the communication board. From there, the communication board uses ethernet to host a GUI web interface and runs SNMP in the background to connect to the facility management systems. The UART sync process is running at a baud rate of 115200 and the protocol is custom. It's 8 bytes at a time including stop and start byte, sensor or controller callouts, and the actual data to transfer.

The controller board started out as an 8 bit CPU when this was supposed to be a simple one time thing. It grew so much that anything past this next upgrade I'm working on now will likely require a whole redesign. I'm likely stuck with the UART this time around but I may try increasing the baud rate to speed things up in that regard.

The other issue I have is the overall architecture. Like stated above, this started out as super simple and just kept growing and getting messier so most of the controller side is polling sensors in loops and then polling buffers for updated commands. Everything is loops checked and updated one by one. This worked when there were only a couple of parameters, but now with hundreds, the update rates are taking way too long. This time around I'm looking for a better way to structure something of this nature along with speed up the UART rate. There's a variable 10-60Hz process that is always running and has the highest interrupt priority because timing is critical. Outside of that, the controller is just polling all parameters one by one and sending the updates to the communication board. A very slow and tedious process.

 

[A.J.710]

I have started using the RP2040 as my default MCU. The UART runs fine at 921,600 Mbs (I believe it supports up to 4 MBps): is that enough for your application?

That should be more than enough. I think I can even boost the speed on my current IC. I think the main issue now is the constant opening and closing of the UART port listening for updates and polling sensors sequentially one by one. Have to see if I can develop some better buffer handling to smooth communication and try to figure out what to do with this polling. Maybe some of the I2C chips can be configured for interrupt triggering or something so I can ease down on these loops.

The problem is on the user side. The system works perfectly fine within the bounds of it's intended use but when users go blowing on environmental sensors and such meant for slow changing conditions, they get upset at the 10 seconds or so the screen takes to show the updated temperature or humidity. In a real world case, the control of the heat exchangers or other devices would not have to react that fast but it's not my job to argue that lol.

 

[Tom.G]

Hardware handshake between the boards using RTS and CTS signals connected to an Priority Interrupt Controller and use DMA to transfer data to/from the communication channel.

The polling that board 1 does could (perhaps) be controlled with a timer connected to one or more high-priority input of the Interrupt Controller, with the inter-board communication relegated to lower priority interrupts.

(RTS = Request-To_Send
CTS=Clear-To-Send
DMA=Direct Memory Access controller)

A bit of a headache to implement though.

A possible variation if Board 1 has Ethernet software routines available, is to use those canned packages that typically have software buffering in their implementation.

BTW, try to avoid writing you own buffered commnication routines if you can; it is difficult to get all the handshaking and timing right for the worst-case possibilities.

Cheers,
Tom

 

[Rive]

The controller board started out as an 8 bit CPU ... I'm likely stuck with the UART this time around but I may try increasing the baud rate to speed things up in that regard.

The issue with the UART is, that while some implementations supports higher baud rate, in general it's not designed for anything fast, especially in older (8bit) controllers.
If your uC can operate the UART and DMA in tandem, so the overhead could be reduced then higher baud rate might help and you need 'only' a full revamp of the method of communication on SW level.

Given the details conveyed so far, I feel this chance slim but worth an investigation anyway.

By the way, did you do any usage/traffic statistics/analysis about that UART? A sniffer might help a lot in identifying the actual circumstances.

 

[berkeman]

Outside of that, the controller is just polling all parameters one by one and sending the updates to the communication board. A very slow and tedious process.

Have you thought about different ways to compress the data, and hence speed up the overall update loop? For example, if you could set update tolerances on each value, the value would not need to be sent on each loop cycle if it did not change outside of its tolerance. You might go ahead and send it anyway every 10 or 20 loops just to show that the sensor is not broken, but if the value is basically not changing by an amount you care about, there probably is no reason to send that data each loop cycle.

 

[harborsparrow]

Back in the 80s, I wrote RT software at Bell Labs for fiber optics comm systems which consisted of as many as 16 different MCUs which needed to work together. We ended up using a shared memory bank to hold options in predetermined addresses. There was a careful analysis and only certain MCUs could write a new value to a given parameter, but every MCU could read any parameter, except that all reads had to be debounced. Debouncing meant taking the reading multiple times rapidly to make sure the values was stable and not in the process of being changed by its master (i.e., allowed-to-write) MCU.

It took a while to convince some team members of the value of using a looped polling scheme to read the shared memory and NO interrupts. Most of the MCUs had to poll, as there was no way to guarantee a lack of deadlocks with interrupts. The software debouncing was used to avoid race conditions.

I don't know if this will be of any use to you. Every option should have a default value at system startup and should only have one master, but all the options are storyboarded for any MCU to poll at whatever interval it can manage.

 

[harborsparrow]

One other possible architecture I once encountered might be of interest to you. I once worked on a radiation monitoring system that delivered an analysis of what-all elements were radiating from a source with what intensity. In such systems, the detector receives so much information so fast that there is no hope of communicating serially. Each specific excitement of the detector was an event of detecting one specific molecule of a specific radiation intensity. What such systems do, then, is count specific events. So one event detecting a molecule of a certain intensity would increment a specific address for that intensity in a shared memory bank to indicate that an event occurred. You don't know exactly when it occurred, but you know how frequently it occurs during a given interval in which detections are being counted.

The immediate output is shown to users as a graph of intensities and event frequency. This data can be analyzed later to determine which specific elements had to be in the studied radiation. PET scanners do this type of thing but with many detectors and then there is also image reconstruction.

It is another form of storyboarding.

The thing about achieving fast performance in any complex system is that you have to design for it from the ground up. Performance generally cannot be retrofitted onto a system. I taught this as a basic tenet of software engineering, though it surely applies to hardware systems as well.

 

[Rive]

The thing about achieving fast performance in any complex system is that you have to design for it from the ground up. Performance generally cannot be retrofitted onto a system.

:sigh:
I've observed a long established routine around that.
me: let's put in some reserves there, additional 50% now is just 1% of the system cost
boss: no need, guys calculated the load, layouters should not meddle
me: OK.
A few months later:
boss: we need some more beef in the system! DO something!
me: that'll be a redesign.

...repeat upon all projects.

 

[Tom.G]

A few months later:
boss: we need some more beef in the system! DO something!
me: that'll be a redesign.

...repeat upon all projects.

Aahhh!
The difference between those that can learn and those that can decree; and never the twain shall meet.