GetFiles and memory problem

Archive
1

I have a problem with my Panda II and .NET 4.1. I have a lot of files in folder and I want read the first one, process it and delete it. In next round I want take another file, process it and delete it and so on. But when I have about 200 files I get out of memory exception when I try get files. The method GetFiles returns string[] with names of files. So if I have 200 files and each has 50 bytes long name (just for example) I need 10000 bytes but I don’t have it. But I don’t need the whole list of files. I need only one. And let’s say I don’t care which one.

Is there a way how to pick up only one file and use only small amount of memory? For example in C++ there are methods like opendir and readdir. I think the combination of both doesn’t use as much memory as GetFiles in C#.

Background:
Why I need this. I want every minute send a package of data to webserver through ethernet. But I want also data when internet connection is not available. So I store data with timestamp on SD card. Something like buffer. And when connection is back again I want send all data to web server. I was thinking about two solutions:

  1. Save package of data in separete files. It is easy to create small file and when connection is back again pickup some file, send it and then delete it. But there is a problem with GetFiles and memory. If the connection is gone all day there would be 1440 files. If set time interval to 5 minute I will get 288 and this is still to much for GetFiles method.
  2. Save all data to one file (Append at the end). But there is problem how to remove processed data. I don’t think there is a method which can remove for example first line in file. Of course I can create such method, but I thought the first option would be much easier and much faster (I think removing first line in 10kB file would be much slower).
2

@ Hall - You could go the second route and store the index into the big file that you have sent. Then once a day (or few hours) you could go write the remaining unsent data to a new file and delete the old one. That shouldn’t be very slow or memory intensive.

You could also look into naming the files a certain way, say 12.txt through 56.txt and keep the start and end index in your program so that you can just open them directly.

Something to consider with the first method is that if you only send a partial message/chunk and the network goes down, you will potentially need to handle that depending on specifics of your application. The second method handles that better because you can just increment the index by that partial amount.

3

@ Hall - I have a better implementation of the GetFiles on codeshare:

https://www.ghielectronics.com/community/codeshare/entry/200

4 Likes
4

Thanks Architect! That is greate solution. Works perfectly…

5

@ Hall - I am glad it helped.

:slight_smile:

6

Great solution for me as well! Using a CerbuinoBee…:clap: