Project - Tiny File System

Archive
1

Tiny File System

Tiny File System provides a file system over any device for which an IBlockDriver can be written. The file system its self is a Log Structured File System which is ideally suited to flash memory devices especially small scale devices which do not have hardware support to wear leveling.

The file system comes with a MemoryBlockDriver which can be used for testing of the interfaces. This driver is based off of non-persistent memory so is not suited for production use.

Public Interfaces:

The file system exposes two classes, one is the TinyFileSystem class and the other is the TinyFileStream. TinyFileSystem provides the necessary functions to Format a device, mount an existing file system from a device, create, open, delete, copy and rename files and more.

The TinyFileStream exposes files created/opened from the file system as a stream. Most stream functionality works as expected. The one exception is that SetLength does not currently allow you to grow the file, this is not a design limitation of the file system, in fact it is quite easy to implement on the current design and will be done eventually.

In addition to the above, the code contains a direct port of the BufferedStream found in the Mono project. The buffered stream is not required for the functionality of the file system, but is useful to optimize writes to the file system. TinyFileStream writes every change directly to the file system which is efficient if you are writing blocks of data at a time, but creates a lot of log entries on the files system which need to be garbage collected. In these scenarios, a BufferedStream can wrap the TinyFileStream to buffer these small writes and write them out in larger chunks.

Whats missing?:

For the most part the file system functionality is pretty much feature complete, except for the ability to expand the file length without actually writing data to the file. The key thing that is still required is a significant amount of testing especially in the scenarios where multiple handles are opened to the same file.

Then of course the primary goal, writing the IBlockDriver implementation for the upcoming Flash memory module from GHI. Unfortunately I have not got the module yet, so this will need to wait until such time as I have the module. In the meantime I want to get this out there so that it can be reviewed by the broader community.

My biggest concern is that of performance, not knowing how the module will perform esp. with some of the decisions I have made with the file system. I have opted to attempt to focus on File System integrity, but there might be a performance cost that is not acceptable so once I have the device and can do real benchmarking I will not make changes in this area unless it will further the goal of FS integrity.

3 Likes
2

Awesome taylorza… :wink:

3

@ Jay Jay, Thank you! It still requires some work, esp. on the code clean-up side, but after 3 days of work over the last two week-ends I thought I wanted to get something out there.

And thanks again for your sort function code share, it saved me time and debugging.

4

@ taylorza - great!

We did ship a module to you right?

5

@ Gus - Yes thank you very much. Unfortunately, living on an Island does not make for speedy delivery but I am sure it will arrive :slight_smile:

6

I have been tinkering a little with the file system and just posted a small update. Some of the changes include the following

  1. Streams are opened as buffered streams by default, no need to explicitly use the BufferedStream.
  2. Buffer size defaults to 4096, but can be overridden. Buffer size of 0 disables buffering.
  3. Added a method to retrieve the available disk space, the reclaimable space (Orphaned clusters).
  4. Added an implementation of the MX2513206BlockDriver***

*** The MX2513206BlockDriver is as yet untested, other than against the emulator component.

7

Thanks for the update.

Mail is really slow this time!

8

It is, I got my Spider within 15 days. I now have two items that have not arrived, the module from you and Architect was also kind enough to send me something more than a month ago that has not arrived.

It might be delayed due to the summer holidays, over-here everything is closed during August, so I am still hopeful…

On a brighter note, I will put up another update today. The FS is now thread safe, so you can write to the FS from multiple threads.

9

The new drop includes the following improvements

  1. Bug fixes based of issues identified using the Flash component emulator
  2. Thread safe access to the file system. Multiple threads reading/writing to the FS concurrently
  3. Potential performance improvements (Requires actual module to confirm - More to come)
10

When is the target date for GHI to release this new module? I am really looking forward to using it in a new design and this Tiny File System code looks perfect! Any chance I can get one now?

11

Unfortunately I have still not received the module which is becoming very disheartening. But I have been sitting on a few ideas to improve the compaction algorithm responsible for reclaiming orphaned clusters when “disk” space becomes low.

This improvement alone has dropped the time my test suite takes to execute a battery of tests from 134 seconds down to 84 seconds, so I am quite excited about that.

12

Ok, so I got impatient and built my own module. It ain’t pretty but should work fine!
Can someone confirm or correct the SPI config below? I am using a FEZ Cerberus, socket 5:

SPI.Configuration cfg = new SPI.Configuration(Cpu.Pin)13), false, 0, 0, false, true, 15000, SPI.SPI_module.SPI2);
SPI spi = new SPI(cfg);

Not sure this is correct. This is my first time doing anything like this, if someone can be so kind as to review and let me know if I am correct or not with setting up the SPI for
this socket and device I’d really appreciate it. Then I can get down to some real testing and post my findings here.

Thanks in advance for any input!

13

@ rpizzo - Sorry I cannot be of any assistance, I still have not received the prototype device.

14

@ taylorza - I got my homemade board working and can write and read using your code :slight_smile: The problem was my SpiConfiguration.

One thing I noticed, I can write a small amount of data to a file, say 100 bytes or less with no trouble, but if I try to write something bigger like > 500 bytes, I throw an exception. I need to investigate this further of course, but I wanted to give you a heads up that the file system and stream and the low level drive code seems to work, at least at first glance.

Oh, one more thing I noticed, on the Dispose() for the TinyfileStream - the base:Dispose() for some reason throws an exception, so that too I need to investigate.

15

@ rpizzo - Thank you for the update, I am glad someone is finally getting to use this code. I still have not received the module and therefore have not been able to test on the real thing.

I would be very interested in what you find regarding the two exceptions. I just re-ran the test suite that I have in which I write large buffers, but again it is not the real device so I could have something that I am missing. As for the exception in the base.Dispose(), what framework are you using, 4.1 or 4.2? I will be 100% ready to support any issues that might arise with this code.

16

Would this be usable on an I2C eeprom with some small modifications or would it be a large effort?

17

@ MikeCormier - I do not know much about working with EEPROMs so I do not know if there are any technical limitations, but the design of the file system is such that it will work anything that can provide an implementation of the IBlockDriver interface though which the File System interacts with the physical device.

The code includes two implementations of IBlockDriver that you can reference.
[ul]MemoryBlockDriver - A driver that is based on a byte array, this was used for the initial implementation and testing
MX25l3206BlockDriver - A driver that implements the specific SPI interface for the GHI Flash module (I wrote an emulator to test this, so I have not run this against the real device)[/ul]

18

@ taylorza - I am using the 4.2 version of the framework. It’s been pretty busy at work these days, this Friday I will devote a large chunk of time to this and provide more detailed data for you. But so far aside from these 2 issues, it’s been smooth sailing!

19

@ rpizzo - that is great, I look forward to your feedback. Thank you for the effort!

20

@ taylorza - I am so sorry for the delay. As fate would have it, out of nowhere I now cannot deploy to my hardware! I can load, build and deploy other solutions, so I know my hardware is all fine. And I can build this solution still with no warnings or errors. But no matter what I try, at deployment time I get:

An error has occurred: please check your hardware.
Object reference not set to an instance of an object.
Source: Microsoft.SPOT.Debugger.CorDebug

This has stumped me for the last 2 days. I even tried remaking my .csproj file, but to no avail.

What little hair I have left is at risk of being all pulled out! Worst case I will start over again. I’ve tried commenting out blocks of code and removing References, it just does NOT want to deploy this solution any more!

If anyone knows exactly what causes this problem, I would be eternally grateful for the answer!