ZeroCM/zcm

Message framing for zcm? #502

DanielArnett posted onGitHub

TL;DR- Given a serial array containing 10 message-types (e.g. a Lat/Lon message, a compass message, and an IMU message), how do you know which bytes correlate with which zcm-defined message types?

@jbendes I love zcm (and the Grand Challenge heritage), thank you for the fantastic documentation. My questions relate to transport on embedded systems. The documentation has made it clear that zcm can effectively serialize/deserialize. My confusion is around how zcm knows which defined message to use when deserializing. Let's say I have a file stored on a hard drive saving 10 different zcm message types in serial: how can I load those messages? Or I have a radio link transmitting/receiving 10 different zcm message types in serial: how do I know which messages to parse, or how to reject noise? I'm aware I may be asking the wrong questions for my problems, that's why I'm reaching out. I feel like there are well-established methods to solve these problems that I'm missing.

I've made home-brewed solutions, but I'm interested in learning the right way, rather than my way. As zcm is built with embedded systems in mind, I was wondering if you've ever done any kind of message framing for transmitting over serial connections.

I can give one example that solves this problem. In Mavlink for instance when the serialized message is sent over a Serial or UDP connection it frames all packets with this kind of structure:

Using Mavlink1 messages for a simple example of message framing. image

Let's pretend I've integrated zcm with mavlink, and I'm using Mavlink's framing structure to send a latitude/longitude I've saved in latlon_t.zcm:

struct latlon_t {
    int32_t lat;
    int32_t lon;
}

Here's an example packet, assuming I created a mavlink message with MSG_ID 0xFE=254 with the above latlon_t:

STX LEN SEQ SYS ID COMP ID MSG ID PAYLOAD CHECKSUM
0xFE 0x08 0x07 0x03 0x02 0xFE 0x 65 AF 40 18 68 B0 DD D3 0x 04 34

Here's how a device receiving this message might process it.

  • STX- "This byte matches the Start-of-text: "0xFE", this must be a mavlink message."
  • LEN- (assuming 8) "This payload will be 8 bytes long."
  • SEQ- "Hmm, the last sequence number was 5, but this sequence number is 7, I must have missed a packet."
  • SYS ID- "This message is coming from vehicle 3."
  • COMP ID- "This message is coming from component 2 on vehicle 3."
  • MSG ID- "Hey this message ID 0xFE matches the one I use for a zcm latlon_t message"
  • PAYLOAD- "Let's unpack the payload to a latlon_t. 8 bytes total, 4 for the lat, and 4 for the lon."
  • CHECKSUM- "The CRC bytes match so I'm 99.998% sure the message is as-sent."

Message framing is valuable when transmitting data over a serial link, as well as when saving/loading data to/from a serial flash memory.

  • Has zcm ever been integrated with any libraries that perform serialization?
  • Am I asking the wrong questions, and do you know of other ways to solve the problem of identifying messages and ensuring accuracy in serial?

Thank you for reading.


@danielarnett has funded $20.00 to this issue.


posted by issuehunt-oss[bot] 3 months ago

I didn't realize IssueHunt required a PR for funds to transfer, I just wanted to donate $20 to the project. Feel free to make a PR to some readme to complete the transfer, or let me know if there's something I can do on my side.

posted by DanielArnett 3 months ago

Hey Daniel! Thanks so much for the enthusiasm! I'm glad you like the project. We love it and it has transferred trillions of events for people in all different use cases. As for embedded support, I think you're probably looking for serial:// transport that zcm provides. This transport handles serialization of zcm onto a linux serial port - handling framing, checksums, synchronization, and message type version checking. That transport relies on a file called generic_serial_transport.{h,c} in the zcm repo if you're interested in the inner workings. This file is then also used on embedded systems and you can take a look at an example of that if you fetch the third party transports submodule of zcm by doing git submodule update --init --recursive, you'll see a new folder zcm/transport/third-party that has a couple examples of implementing a serial transport on a bare metal embedded system. Let me know if you have any more questions!

posted by jbendes 3 months ago

Ah, yes generic_serial_transport.c does describe the framing!

// Framing (size = 9 + chan_len + data_len)
//   0xCC
//   0x00
//   chan_len
//   data_len  (4 bytes)
//   *chan
//   *data
//   sum1(*chan, *data)
//   sum2(*chan, *data)

To continue my above example table of a lat/lon message. I'll assume the channel name is "latlon", and the data is still 4 bytes for each number.

STX CHAN_LEN DATA_LEN CHAN DATA CHECKSUM
0x CC 00 0x06 0x08 "latlon" 0x 65 AF 40 18 68 B0 DD D3 0x 04 34
posted by DanielArnett 3 months ago

Fund this Issue

$20.00
Funded

Pull requests

Recent activities

danielarnett funded 20.00 for ZeroCM/zcm# 502
3 months ago