Project - Tiny File System

Architect · September 23, 2013, 6:27pm

That would be great!

Simon_from_Vilnius · September 24, 2013, 6:23am

I think I found a bug. Filename length is limited to 16 symbols, but it is not checked on file creation, and file is created probably overwriting something. For example:


 for (int i = 0; i < 12; i++) {
     using (var myFile = myTinySystem.Create("lengthyfile" + i + ".txt")) { }
}

foreach (string file in myTinySystem.GetFiles()) {
     Debug.Print("File: " + file);
}

Results in:



File: LENGTHYFILE0.TXT
File: LENGTHYFILE1.TXT
File: LENGTHYFILE10.TX"
File: LENGTHYFILE2.TXT
File: LENGTHYFILE3.TXT
File: LENGTHYFILE4.TXT
File: LENGTHYFILE5.TXT
File: LENGTHYFILE6.TXT
File: LENGTHYFILE7.TXT
File: LENGTHYFILE8.TXT
File: LENGTHYFILE9.TXT

Note the LENGTHYFILE10.TX". That scrambles the filesystem and it starts crashing on most of the functions.

taylorza · September 24, 2013, 7:08am

@ Simon from Vilnius - Thank you for the feedback, I will upload an updated version.

Simon_from_Vilnius · September 24, 2013, 9:16am

Another glitch, this time — a bad one. Lets call WriteAllBytes repetitively:


            myTinySystem.Format();
            myTinySystem.Mount();

            myTinySystem.WriteAllBytes("filename.dat",Encoding.UTF8.GetBytes("Labas pasauli!"));
            Debug.Print("filename.dat exists: "+(bool) myTinySystem.Exists("filename.dat"));
            Debug.Print("Number of files: "+ myTinySystem.GetFiles().Length);

            myTinySystem.WriteAllBytes("filename.dat", Encoding.UTF8.GetBytes("Labas pasauli!"));
            Debug.Print("filename.dat exists: " + (bool)myTinySystem.Exists("filename.dat"));
            Debug.Print("Number of files: " + myTinySystem.GetFiles().Length);

            myTinySystem.WriteAllBytes("filename.dat", Encoding.UTF8.GetBytes("Labas pasauli!"));
            Debug.Print("filename.dat exists: " + (bool)myTinySystem.Exists("filename.dat"));
            Debug.Print("Number of files: " + myTinySystem.GetFiles().Length);

            myTinySystem.WriteAllBytes("filename.dat", Encoding.UTF8.GetBytes("Labas pasauli!"));
            Debug.Print("filename.dat exists: " + (bool)myTinySystem.Exists("filename.dat"));
            Debug.Print("Number of files: " + myTinySystem.GetFiles().Length);

            myTinySystem.WriteAllBytes("filename.dat", Encoding.UTF8.GetBytes("Labas pasauli!"));
            Debug.Print("filename.dat exists: " + (bool)myTinySystem.Exists("filename.dat"));
            Debug.Print("Number of files: " + myTinySystem.GetFiles().Length);

            myTinySystem.WriteAllBytes("filename.dat", Encoding.UTF8.GetBytes("Labas pasauli!"));
            Debug.Print("filename.dat exists: " + (bool)myTinySystem.Exists("filename.dat"));
            Debug.Print("Number of files: " + myTinySystem.GetFiles().Length);

Results in:


filename.dat exists: True
Number of files: 1
filename.dat exists: False
Number of files: 1
filename.dat exists: True
Number of files: 2
filename.dat exists: False
Number of files: 2
filename.dat exists: True
Number of files: 3
filename.dat exists: False
Number of files: 3

WriteAllBytes creates ghosts! Will check ReadAllBytes now

taylorza · September 24, 2013, 9:53pm

@ Simon from Vilnius - I have updated with code with the test for the file length when the file is created.

I also fixed the issue you found with the WriteAllBytes, there was a bug in the Truncate method which is called by WriteAllBytes. That was a good find, thank you!

Please let me know if you find any other issues, I will be happy to fix any bugs.

taylorza · September 24, 2013, 9:59pm

@ Gus, the fixes I have posted would also affect the Gadgeteer Module driver I wrote for the Flash Module. How can I update this?

The original content with the link has been moved from the wiki to the GHI documentation so I can no longer update the content that I created, is there a process for this?

Simon_from_Vilnius · September 25, 2013, 2:40am

Thank for the quick fixes! Appears to be working! However, I’ve got another bug for you


            myTinySystem.Format();
            myTinySystem.Mount();

            myTinySystem.WriteAllBytes("small.txt", Encoding.UTF8.GetBytes("Small letters"));
            myTinySystem.WriteAllBytes("SMALL.txt", Encoding.UTF8.GetBytes("BIG LETTERS"));


            Debug.Print("Content of small.txt: "+new string(Encoding.UTF8.GetChars(myTinySystem.ReadAllBytes("small.txt"))));
            Debug.Print("Content of SMALL.txt: " + new string(Encoding.UTF8.GetChars(myTinySystem.ReadAllBytes("SMALL.txt"))));

Output:


Content of small.txt: BIG LETTERS
Content of SMALL.txt: BIG LETTERS

Looks like creating files is case-sensitive, but reading — is not…

taylorza · September 25, 2013, 7:32am

Simon,

The file system is not case sensitive, so when you WriteAllBytes(“small.txt”,…) and then WriteAllBytes(“SMALL.TXT”,…) the second call overwrites the first file.

When you read the file, both reads are reading the same file, ie the last file that was written with the data in uppercase.

If you call myTinySystem.GetFiles() after the two calls to WriteAllBytes, you should see that you only have one file.

Simon_from_Vilnius · September 25, 2013, 12:01pm

Sorry, my bad. You’re right for this example. It works with plain old good latin characters.

My initial problem was with unicode characters like “š”. It is possible to create two files “š.txt” and “Š.txt”; I guess ToUpper() function is not fully Unicode-compatible. Can we do something about it or should you just put a note in TinyFileSystem description?..

taylorza · September 30, 2013, 7:17pm

@ Simon from Vilnius - I took a quick look at the .NETMF implementation of ToUpper/ToLower and sadly it seems that they are not catering to Unicode characters at all, the case change only applies to the Latin characters.

For your enjoyment, here is the relevant code

https://netmf.codeplex.com/SourceControl/latest#client_v4_3/CLR/Libraries/CorLib/corlib_native_System_String.cpp


    for(CLR_UINT32 i=0; i<arrayTmp->m_numOfElements; i++)
    {
        CLR_UINT16 c = *ptr;

        if(fToUpper)
        {
            if(c >= 'a' && c <= 'z') c += 'A' - 'a';
        }
        else
        {
            if(c >= 'A' && c <= 'Z') c -= 'A' - 'a';
        }

        *ptr++ = c;
    }

Simon_from_Vilnius · October 1, 2013, 2:39am

Hmmm… Ok, not a big problem. I think you should update description with a few notes:

File name is limited to 16 [em]bytes[/em], that is, to 16 plain latin characters or to less that 16, if there’s anything like “ąčę” present;
Non-latin characters in file name should be avoided.

Simon_from_Vilnius · October 1, 2013, 2:44am

Another question. As an author of the TinyFileSystem. do you have any ideas of the optimal sector size for FRAM? I now use default ones (cluster 256, sector 1024), but his generates orphaned bytes quite quickly. I’ve tried 16-64, (doesn’t work) and 64-256 (works), but still don’t know which configuration to use. Could there be any performance issues depending on configuration?

taylorza · October 1, 2013, 11:19pm

There is no one right answer to this question, the trade-offs are the same or similar to the trade-offs for the cluster size used on file systems like FATxx, NTFS etc.

Some general guide lines would be

[ul]The cluster size must be larger that 32 bytes, the file header (on the first cluster of the file) is 32 bytes in size.
If you create many small files then you might want smaller clusters so that you do not waste large clusters on small files. A cluster is the smallest unit of allocation for the file system, creating a file with just one byte would require a full cluster.
If you create large files then bigger clusters might be better because you can offset the overheads of the cluster headers. Especially if you are continually appending to the file.[/ul]
You would need to test to see what works best for the size of the device you are using and the type of files you are creating. Also to minimize the number of orphaned clusters you should use the buffered stream option.

taylorza · October 1, 2013, 11:24pm

Done. I updated the description in the code share. Thank you.

Simon_from_Vilnius · October 14, 2013, 2:12am

I’ve found a strange bug. My FRAM tinyFileSystem has stopped working. If I call any function related to file (.GetFiles(), for example), an Exception “Past end of address space” is thrown. However, if I call .GetStats() function, I get:

>? _framFilesystem.GetStats()
{Bytes Free: 254976
Bytes Orphaned: 0}
    BytesFree: 254976
    BytesOrphaned: 0

So my files are in the chip, taking a few kBytes, but I cannot use them.

I’ve ran into this several times already, and formatting the filesystem surely helps. But I do not want to format now. What could I do to pinpoint the bug?

taylorza · October 14, 2013, 7:32pm

@ Simon from Vilnius - Is there a sequence of steps that can reproduce the problem? GetFiles will just read the first cluster of the file and extract the filename from that.

Based on the error you are getting, it sounds like the cluster id is incorrect and pointing to a cluster that is outside of the address space.

The function GetFiles() calls GetFileName() internally, can you check the cluster id that is used.


    private string GetFileName(FileRef file)
    {
      // Extract the file name from the first block of the file
      // which is a File Cluster.
      ushort clusterId = file.Blocks[0]; // <--------------- CHECK CLUSTER ID
      _blockDriver.Read(clusterId, ClusterBuffer.FileNameLengthOffset, _cluster, ClusterBuffer.FileNameLengthOffset, 2 + ClusterBuffer.MaxFileNameLength);
      return _cluster.GetFileName();
    }

Basically the file.Blocks array is an array of the clusters that contain the files content, so you can check all the cluster ids to see if they are reasonable or would exceed the file system size.

I have an emulator that I wrote for the file system, so if you can send me the device metrics (size, cluster size etc.) that you use, I can try simulate the problem on the emulator. It would really help if you have some sample code that would reproduce the issue.

Simon_from_Vilnius · October 15, 2013, 8:21am

file.Blocks: its contents are 1023,0,1,2,3,4,5,6,11,12,9,10. Looking suspicious, but not necessarily wrong…

Reproducing is the problem I just write, read, write read, and sometimes it just locks up. I can’t provide you the exact steps…

Simon_from_Vilnius · October 15, 2013, 8:28am

I was digging in a bit… When

_blockDriver.Read(clusterId, ClusterBuffer.FileNameLengthOffset, _cluster, ClusterBuffer.FileNameLengthOffset, 2 + ClusterBuffer.MaxFileNameLength);

executes, it goes to

 int address = (clusterId * ClusterSize) + clusterOffset;
            if (address + data.Length> DeviceSize) { //<--- shouln't it be address + count?
                throw new Exception("Past end of address space"); 
            }
            var resultBytes = Read(address, address + count);
            Array.Copy(resultBytes, 0, data, index, count);

Check my comment. In my case, clusterId is 1023, and total number of clusters is 1024, so address + data.Length always goes beyond the maximum address! I changed it to address + count, and now I can access the same “corrupted” file again

This also explain the rarity of the bug. It only happens when the last sector of the filesystem is involved.

taylorza · October 15, 2013, 5:38pm

@ Simon from Vilnius - You are correct, the same error exists in the Write method. I have fixed it and will update the code share shortly.

I am surprised you hit this, it is an issue in the MemoryBlockDriver only, the physical device block driver (MX25l3206BlockDriver) does not have this issue. Are you actually using the MemoryBlockDriver for something?

I could not understand why my unit tests did not hit this case, but thanks to you tracking it down I see the issue is that I am using a SPI emulator that I wrote which works like the physical device and the physical driver does not have this issue. I will need to review the MemoryBlockDriver.

Thanks again for helping to make Tiny File System better! It is truly appreciated.

Simon_from_Vilnius · October 16, 2013, 1:33am

I am using a FRAM chip, and since it is very similar to RAM, I used MemoryBlockDriver as a starting point. That explains how the bug got through to my driver implementation. A few more bugs and it may be ready for public use