Hydra SD Performance

Archive
1

What is the maximum data throughput of the Hydra via GHI’s SD card module?

The reason I ask is that I’ve implemented a data logger which receives 80 byte packets via an incoming serial port at 100Hz and at 115200 baud and then logs 100 byte packets to the SD card at a user selected logging rate.

Typically at a 20Hz logging rate, i.e. writing out every 5th incoming packet I don’t see any issues. However at higher logging rates like 50Hz or 75Hz I often see the following IO exceptions a couple of seconds after data logging starts.


IO exception
    #### Exception System.IO.IOException - CLR_E_FILE_IO (6) ####
    #### Message: 
    #### Microsoft.SPOT.IO.NativeFileStream::Write [IP: 0000] ####
    #### System.IO.FileStream::Write [IP: 002a] ####
    #### FTIHydraLogger.Program::RecordData [IP: 0017] ####
    #### FTIHydraLogger.Program::StartLogging [IP: 00cb] ####
A first chance exception of type 'System.IO.IOException' occurred in Microsoft.SPOT.IO.dll

There’ll be a flood of about 30 of these exceptions and then the system freezes. The exception info doesn’t give any details on the actual nature of the file IO error.

Invariably the SD card will then need a chkdsk and often a format before it becomes usable in the Hydra again.

20Hz @ 100bytes ~ 2KB/s
50Hz @ 100bytes ~ 5KB/s

I’m assuming the maximum throughput surely can’t be as low as somewhere between 2 - 5KB/s?

I’m using the following code to write out the data, i.e. a single call to Write() with a 100byte buffer.


void RecordData()
{
	if (_dataLogFile != null)
	{
		try
		{
			_dataLogFile.Write(_data.GetBuffer(), 0, Data.RAW_PACKET_SIZE);

			if (_fileFlushingTimer.PeriodElapsed(_fileFlushingPeriod))
				_dataLogFile.Flush();
		}
		catch (Exception ex)
		{
			Debug.Print("IO exception");
		}
	}
}

Thanks

2

Is it a multithreaded application?

3

You sure you arent opening a file thats already open? I think thats what architect is asking :o)

Instead of doing it every 100 bytes why dont you write every 10 seconds or at a different timed interval. For example store all the incoming data in memory, every 10 seconds open the file and write the data, clear the memory and start again. This will prevent you trying to open a file before it been released, at higher bitrates you might find your program is opening the file before the close has completed which causes the IO error.

Also double check any error handling in your file write function ensures that any opened files have been closed. You dont want to leave a file open and in limbo :o)

Paul

4

There is a single thread created during ProgramStarted which does all the data logging, i.e. retrieving incoming serial data, sampling some analog and digital ports and then writing out the 100byte packet to the SD card.

The main thread which calls ProgramStarted is only used to service a digital input interrupt, so it’s not the case that multiple threads are contending for SD IO access etc.

5

No the file _dataLogFile is only opened once during startup in ProgramStarted before the data logging thread is created. So during logging the file is already open and it’s simply issuing a Write() call on a 100byte buffer.

What sort of IO throughput in this sort of scenairo have others seen with Hydra and GHI’s SD card module?

6

Is it a short bit of code that you can post?

I am worried that you are opening the file in one function and then writing to it in another after its been release by garbage collection. Either that or It also sounds to me like your opening the file in one thread and then writing to it from another thread which could cause issues. Try opening the file in the same thread that your writing from if thats possible :o)

Paul

7

@ Bodwad -

The _dataLogFile is a member of the Program class, so it never goes out of scope. It is created and the file opened by the main thread during the call the ProgramStarted and before the data logging thread is created and started.

The RecordData() method is called from the data logging thread at a user defined rate, e.g. 20Hz, 50Hz etc.

The main thread never uses _dataLogFile after creating it, the only access to it is the call to Write() from the data logging thread, so there is no thread contention over _dataLogFile.

As I mentioned at lower data rates, e.g. 20Hz ~ 2KB/s and close to 50Hz ~ 5KB/s everything works fine and it will log for hours without seeing these IO exceptions.

But pretty much anything over 50Hz then starts showing these IO exceptions.

8

@ Sean - I don’t remember exactly, but I think Wouter did some tests and posted performance numbers related to SD on the forum.
Although it might be only for his SD RLP code.