I am having problems with logging data on a thumb drive. I wrote under another topic yesterday and was suggested to create a new one so here it is. Just a summary of what I am trying to do accompanied with the actual code block.
Acquire sensor data using a custom sensor board. Total packet size is 62 bytes and packet update rate is 50 Hz.
Send the data to FEZ domino via SPI. Here I am sending the 62 byte packet every 20ms. No problems in receiving it.
Accumulate the packets for a second. This makes the total data size to 3100 bytes. I have a circular buffer implemented in FEZ. Its a 6200 bytes array with two pointers one for writing and the other for reading. Its a well tested logic and I have done detailed debugging. There is nothing wrong over here either.
Once I have more than 25 packets i.e. 1 sec of data, I write it onto the USB thumb drive. This is where the problem occurs. If I comment the line of code where I send 3100 bytes to USB thumb drive, every thing works fine. I don’e get any packet miss issues. Again I have done detailed debugging and found no packet miss issues. But when I uncomment this line of code, FEZ just doesn’t seem to be able to keep up. I have pasted the code chunk performing the above mentioned tasks.
One more thing. I have checked two different USB thumb drives even an SD card (as SD card has a 4bit interface so I thought this will help) but no luck.
I am not saying that what the designers are claiming (500,000 bytes per second) is incorrect. I am sure they performed a thorough benchmarking before claiming this. There is something wrong with my coding. Any suggestions/ advice will be appreciated.
I think you’re better of setting up 2 threads instead of the endless loop. One thread will be reading the sensor and setting a flag when you have 1 sec of data accumulated. The second thread takes care of writing the data to the thumb drive depending on the flag you set in the 1st thread.
I am a total newby on Fez related stuff, but I am quite good in software… I see you implemented a double buffering technique, but that is useless if you don’t use separate threads to read and write the buffers. From what I get from the code you are reading a number of readings, and then the reading of the sensors is being ‘blocked’ by them to storage. The actual writing of the buffer is what takes some time (seeking, writing, validating) but certainly more than a couple of msecs. Check out the threading possibilities of the netmf and you will gain a substantial performance boost.
Thanks for the suggestions.
@ Eric , RobV. I have already tried the 2 thread approach. Same issues in this case. I have pasted the code below. See if I am missing something. In this thread I just check if I have 1 sec of data. If yes then write it on USB and sleep for 20 ms. This thread has a lower priority then the main thread responsible for receiving and buffering data.
Does any one know the inner working of FileStream.write function. Does it create a buffer of its own and then send data to USB when ever it gets the time slot or it just blocks every thing until all of the data is written. I am asking this question because I have noticed that if I send fewer bytes (124 bytes every second i.e just two packets and discard the rest), there is no problem with my code. But as soon as I start going over this limit, I end up in the same ditch.
You may want to move the FEZ_ACK flag to after the file data has been written or till you know you are not going to write the data.
The write speed is very fast but that is the average write speed. There are times where write finishes in 1ms and others where it can take 200ms or even more. Why? Depending on what is going on in the file system structure…are we looking for free clusters… clusters is ready…need to flush or we have room for buffering…etc.
Also, tt is better if you just write small chunks to file system…not too small though. Why? Because you want to the the file system take care of buffering instead of you managing it.
What is the age or speed rating of the memory card you are using? Have you checked with another memorycard? Maybe you could attach the full sourcecode so see how the threads are managed. Also remember that when using multiple threads you need to code synchronize at some point because your 2 threads are basically sharing the same variables.
Can you dump random data to your USB stick in a tight loop and tell me how many bytes it is writing in a second? Write 1 million bytes and tell me how long it took
@ RobV. Good suggestions. The thumb drive is a fairly new one. I tried two different thumb drives and even an SD card. I don’t think my two threads ever end up accessing same memory location as my read and write pointers are never the same (Always keeping the read pointer behind the write one). Write pointer is managed in main thread where as read pointer in the USB_WRITE thread.
@ Gus. Will do this benchmarking with the thumb drives I have to get some timing statistics. I will publish the results on this topic.
I have managed to make it work for now. Instead of buffering the data at FEZ end, I am buffering it on my sensor board for 1 second. Then sending a burst of data to FEZ via SPI (now running at 10 MHz) and writing it to the thumb drive. I tested this logic the whole night (8 hours) and not a single packet is lost. So FEZ is capable of writing 3100 bytes on my thumb drive every second if nothing else needs to be done at this end.
My guess is you have this problem, while FEZ is taking care of the file system, it may end up needing to do more work to handle the FAT table. Like I said before, the average speed is pretty fast but you still need to have some enough buffering so if the file system operation took 200ms then you will have no problems.