Cobra III - logging CAN to USB

Hi,

I am looking for any pointers on the quickest way to get a CAN message written to a USB file. I have 1 or 2 CAN buses running at 500kbps, and I am trying to log as much as possible. If I filter the messages down using Group or Explicit filters, I can keep pace while writing to the USB, either as raw data (to be converted later), or by converting to standard ASC format that can be loaded in to various industry-standard packages.

My main issue is if I try and log a full bus, and I can’t write as quick as the messages come in.

I have experimented with a lot of options, and writing to a Filestream 1-2K at a time (buffered manually or using the buffer option in the constructor) seems to be the best in my setup. StreamWriter is slower, and writing strings is much slower than writing bytes. If I simply write a dummy byte set (21 bytes to cover channel, timestamp, Arbitration ID and data bytes), then the buffer does not fill up. However, as soon as I start to populate those bytes from actual CAN messages, the buffer fills up quicker than I can process it. I thought the USB access would be the bottleneck, but it seems to be the CAN message retrieval

I have tried all sorts of ways to get the data out of the CAN.Message structire in to a byte array and then write the byte array. e.g. for the 8 data bytes, I have tried Array.Copys, for-loops, and even played with pointers, but I can’t get it quick enough.

I was wondering if there was a way to avoid retrieving the various parts of the CAN.message structure individually and simply copy the raw CAN.message data (as bytes) direct to the USB file, to try and be as efficient as possible? It may have a few extra bytes I don’t need (IsExtendedID etc) but the speed may outweigh that.

Is there some way, pointers or otherwise, to grab the CAN data at source, either as messages, or perhaps as an incoming byte stream?

Any help or suggestions much appreciated.

Nick

CAN can be very fast. Do you must have all data recorded? Is data coming in bursts that it can be buffered for later processing? Maybe a native RLP driver to grab CAN data and cook the data and then only do the recording on the managed side.

Hi Gus,

Apologies in advance for the length of this reply!!

I am trying to make a data logger, so an option to log all CAN is the ultimate aim. For the buses I am logging, it is 500kbps, with all frames 8 bytes, running at 2200-2400 frames/second pretty consistently. There is a slight waxing and waning due to the scheduler, but they are basically all periodic frames, so it doesn’t vary much. I have set the HW buffer to 10,000 messages, but they key thing is that the buffer is filling up, so it will always max out at some point however high I set the buffer.

In tests, reading multiple messages at a time with ReadMessages() was faster than reading them individually with ReadMessage() (not surprisingly!), and I have tried various combinations of number of messages to read at a time, and bytes to write to file at a time. At the moment, writing 1-2K to the file at a time, and reading 100 messages at a time to process seems to be about the best performance.

As you hint, getting the data from the CAN in to a byte array to write to a file seems to be the longest part. If I just retrieve the messages, don’t touch them, but write a dummy byte array for each message (to simulate everything but the CAN data extraction itself), my queue of incoming messages does not increase, and stays below 100. As soon as I start to do any transfer of data from CAN to the file-write buffer, then I can’t keep up with incoming CAN. At 2000+ frames/second, I need to be able to grab the CAN timestamp, ArbID and 8 bytes of data in less than 0.5ms, leaving enough time for writes to file every 50-100 messages or so.

Are there any examples of RLP for CAN? I have dabbled with RLP only once before when trying to speed up writes to a display but it was very basic indeed.

Thanks

Nick

For interest, my main test loop currently looks like this…

                  msgsReceived = loggerCAN1.ReadMessages(CAN1RxMsg, 0, CAN1RxMsg.Length);
                for (int i = 0; i < msgsReceived; i++)
                {
                    PrintDebugtime("Checking msg " + i.ToString());
                    // Copy CAN info in to Byte Array
                    // 8 byte timestamp
                    Array.Copy(BitConverter.GetBytes(CAN1RxMsg[i].TimeStamp.Ticks), 0, CANLogByteArray, (i*21), 8);
                    PrintDebugtime("After timestamp");

                    // 1byte channel
                    CANLogByteArray[(i*21) + 8] = 1;
                    PrintDebugtime("After channel");

                    // 4 byte ArbID
                    Array.Copy(BitConverter.GetBytes(CAN1RxMsg[i].ArbitrationId), 0, CANLogByteArray, (i*21) + 9, 4);
                    PrintDebugtime("After ArbID");
                    
                    // 8 bytes of data
                    Array.Copy(CAN1RxMsg[i].Data, 0, CANLogByteArray, (i * 21) + 13, 8);
                    PrintDebugtime("After data bytes");
                }
                PrintDebugtime("About to write");
                CANFileLog2.Write(CANLogByteArray, 0, msgsReceived * 21);
                PrintDebugtime("Just written");

With the rough-and-ready debug time routine :

    public void PrintDebugtime(string DebugText)
    {
        Debug.Print(DebugText + "-" + (Microsoft.SPOT.Hardware.Utility.GetMachineTime().Ticks - oldTicks).ToString());
        oldTicks = Microsoft.SPOT.Hardware.Utility.GetMachineTime().Ticks;
    }

It gives a flavour of the timing (ignoring the fact that it takes time to do the debug print itself, typically :


Checking msg 98-9656
After timestamp-3664
After channel-1748
After ArbID-3199
After data bytes-2661
Checking msg 99-9636
After timestamp-4665
After channel-1757
After ArbID-3142
After data bytes-2671
About to write-1660
Just written-267221

I am not sure yet why it takes 9636 ticks to go from writing data bytes for message 98 to the “Checking msg 99” as that should be almost the next command as it restarts the loop. Does a for loop take that long to evaluate? Dropping out of the loop takes only 1660 ticks to get to the “About to write” print as a counterpoint…
You can see that frame 99 takes 4665+1757+3142+2671 = about 12,200 ticks to copy the data over (about 1.2ms per message). The write to file of 21 bytes *100 takes 267221 ticks (27ms), so about 0.25ms per message.

My head is swimming with all the different variables I have tried adjusting and ways to move bytes, so I hope I am not missing something obviously slowing it down…

Is using SD an option?

Certainly is an option - not quite as handy overall but certainly an option! That would only speed up the writing though. It wouldn’t speed up getting the CAN data out…

Is SD access typically faster than USB? We used SD on a Panda II, but I never speed tested it…

Nick

If SD is an option then consider our newer hardware with TinyCLR. Adding native drivers is a lot more flexible there. Plus we can request changes to the CAN drivers as it is still in development.

Also, if you are working on a commercial design, you should have a discussion with our sales. They will give you tips on the available options.

1 Like

Thanks Gus,

This is initially additional functionality on existing Cobra III-based hardware. I understand that TinyCLR will work on Cobra III, but CAN was not added last time I checked. Is TinyCLR likely to be faster on the same hardware for this sort of thing? Does TinyCLR not do USB as well (just wondering why you mentioned SD specifically)?

We may make a standalone hardware for this, so I will explore the other options as well.

Thanks

Nick

TinyCLR is an alive product in full swing. NETMF is discontinued and not supported.
TinyCLR has CAN today and the whole OS is getting improvements daily.
USB Host is not in TinyCLR yet and it will be months before it is there but SD is already there.
Finally, FEZ Cobra III is also obsolete so here is another reason to discuss your needs with our team.

Hi,

Quick speed tests show the SD access (in NETMF) is much slower, so I need to stay with USB, which means no chance to move to TinyCLR yet.

The GHI website lists G120 SoM as active and maturing in 2027. I know it is not clear whether that applies to G120 extension products such as the Cobra III, and what constitutes software support, but for now we need to stick to this product, so I will persevere with NetMF for this, at least until USB Host comes to TinyCLR. At that point, I can investigate whether I can move parts from NetMF to TinyCLR using in-field update.

Back to the original problem - is there any way (from NetMF) to access the raw data in the CAN message directly, rather than using the specific methods for each part? Is it possible to do something like copy XX bytes starting at the memory pointer for a specific CAN message?

Thanks

Nick

No raw bytes in NETMF unfortunately.

Take a look at this, for the future when you use TinyCLR, Using native code

Thanks Gus. I’ll do some homewotk ready for when TinyCLR supports all the features we need.

Nick