Main Site Documentation

Ftp download class using socket and speed problems


#1

hi,
i have implemented an ftp downloader class from the source code of Jaimon Mathew .
i just ported login and download functions , so you can implement other features :slight_smile:


using System;
using System.IO;
using GHIElectronics.NETMF.Net.Sockets;
using Microsoft.SPOT;
using GHIElectronics.NETMF.Net;
using System.Text;
using System.Threading;



namespace smg.ftp
{

    public class FtpClient
    {

        public class FtpException : Exception
        {
            public FtpException(string message) : base(message) { }
            public FtpException(string message, Exception innerException) : base(message, innerException) { }
        }

        private static int BUFFER_SIZE = 2048;// 512;
        private static Encoding ASCII = Encoding.UTF8;

        private bool verboseDebugging = false;

        // defaults
        private string server = "localhost";
        private string remotePath = ".";
        private string username = "anonymous";
        private string password = "anonymous@ anonymous.net";
        private string message = null;
        private string result = null;

        private int port = 21;
        private int bytes = 0;
        private int resultCode = 0;

        private bool loggedin = false;
        private bool binMode = false;

        private Byte[] buffer = new Byte[BUFFER_SIZE];
        private Socket clientSocket = null;

        private int timeoutSeconds = 300;

        /// <summary>
        /// Default contructor
        /// </summary>
        public FtpClient()
        {
        }
        /// <summary>
        /// 
        /// </summary>
        /// <param name="server"></param>
        /// <param name="username"></param>
        /// <param name="password"></param>
        public FtpClient(string server, string username, string password)
        {
            this.server = server;
            this.username = username;
            this.password = password;
        }
        /// <summary>
        /// 
        /// </summary>
        /// <param name="server"></param>
        /// <param name="username"></param>
        /// <param name="password"></param>
        /// <param name="timeoutSeconds"></param>
        /// <param name="port"></param>
        public FtpClient(string server, string username, string password, int timeoutSeconds, int port)
        {
            this.server = server;
            this.username = username;
            this.password = password;
            this.timeoutSeconds = timeoutSeconds;
            this.port = port;
        }

        /// <summary>
        /// Display all communications to the debug log
        /// </summary>
        public bool VerboseDebugging
        {
            get
            {
                return this.verboseDebugging;
            }
            set
            {
                this.verboseDebugging = value;
            }
        }
        /// <summary>
        /// Remote server port. Typically TCP 21
        /// </summary>
        public int Port
        {
            get
            {
                return this.port;
            }
            set
            {
                this.port = value;
            }
        }
        /// <summary>
        /// Timeout waiting for a response from server, in seconds.
        /// </summary>
        public int Timeout
        {
            get
            {
                return this.timeoutSeconds;
            }
            set
            {
                this.timeoutSeconds = value;
            }
        }
        /// <summary>
        /// Gets and Sets the name of the FTP server.
        /// </summary>
        /// <returns></returns>
        public string Server
        {
            get
            {
                return this.server;
            }
            set
            {
                this.server = value;
            }
        }
        /// <summary>
        /// Gets and Sets the port number.
        /// </summary>
        /// <returns></returns>
        public int RemotePort
        {
            get
            {
                return this.port;
            }
            set
            {
                this.port = value;
            }
        }
        /// <summary>
        /// GetS and Sets the remote directory.
        /// </summary>
        public string RemotePath
        {
            get
            {
                return this.remotePath;
            }
            set
            {
                this.remotePath = value;
            }

        }
        /// <summary>
        /// Gets and Sets the username.
        /// </summary>
        public string Username
        {
            get
            {
                return this.username;
            }
            set
            {
                this.username = value;
            }
        }
        /// <summary>
        /// Gets and Set the password.
        /// </summary>
        public string Password
        {
            get
            {
                return this.password;
            }
            set
            {
                this.password = value;
            }
        }

        /// <summary>
        /// If the value of mode is true, set binary mode for downloads, else, Ascii mode.
        /// </summary>
        public bool BinaryMode
        {
            get
            {
                return this.binMode;
            }
            set
            {
                if (this.binMode == value) return;

                if (value)
                    sendCommand("TYPE I");

                else
                    sendCommand("TYPE A");

                if (this.resultCode != 200) throw new FtpException(result.Substring(4));
            }
        }
        /// <summary>
        /// Login to the remote server.
        /// </summary>
        public void Login()
        {
            if (this.loggedin) this.Close();

            Debug.Print("Opening connection to " + this.server + "FtpClient");


            IPEndPoint ep = null;

            try
            {
                this.clientSocket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
                ep = new IPEndPoint(IPAddress.Parse(this.server), this.port);
                this.clientSocket.Connect(ep);
            }
            catch (Exception ex)
            {
                // doubtfull

                if (this.clientSocket != null) this.clientSocket.Close();

                throw new FtpException("Couldn't connect to remote server", ex);
            }

            this.readResponse();

            if (this.resultCode != 220)
            {
                this.Close();
                throw new FtpException(this.result.Substring(4));
            }

            this.sendCommand("USER " + username);

            if (!(this.resultCode == 331 || this.resultCode == 230))
            {
                this.cleanup();
                throw new FtpException(this.result.Substring(4));
            }

            if (this.resultCode != 230)
            {
                this.sendCommand("PASS " + password);

                if (!(this.resultCode == 230 || this.resultCode == 202))
                {
                    this.cleanup();
                    throw new FtpException(this.result.Substring(4));
                }
            }

            this.loggedin = true;

            Debug.Print("Connected to " + this.server + "FtpClient");


        }

        /// <summary>
        /// Close the FTP connection.
        /// </summary>
        public void Close()
        {
            Debug.Print("Closing connection to " + this.server + "FtpClient");

            if (this.clientSocket != null)
            {
                this.sendCommand("QUIT");
            }

            this.cleanup();
        }



        /// <summary>
        /// Return the size of a file.
        /// </summary>
        /// <param name="fileName"></param>
        /// <returns></returns>
        public long GetFileSize(string fileName)
        {
            if (!this.loggedin) this.Login();

            this.sendCommand("SIZE " + fileName);
            long size = 0;

            if (this.resultCode == 213)
                size = long.Parse(this.result.Substring(4));

            else
                throw new FtpException(this.result.Substring(4));

            return size;
        }


        /// <summary>
        /// Download a file to the Assembly's local directory,
        /// keeping the same file name.
        /// </summary>
        /// <param name="remFileName"></param>
        public void Download(string remFileName)
        {
            this.Download(remFileName, "", false);
        }

        /// <summary>
        /// Download a remote file to the Assembly's local directory,
        /// keeping the same file name, and set the resume flag.
        /// </summary>
        /// <param name="remFileName"></param>
        /// <param name="resume"></param>
        public void Download(string remFileName, Boolean resume)
        {
            this.Download(remFileName, "", resume);
        }

        /// <summary>
        /// Download a remote file to a local file name which can include
        /// a path. The local file name will be created or overwritten,
        /// but the path must exist.
        /// </summary>
        /// <param name="remFileName"></param>
        /// <param name="locFileName"></param>
        public void Download(string remFileName, string locFileName)
        {
            this.Download(remFileName, locFileName, false);
        }

        /// <summary>
        /// Download a remote file to a local file name which can include
        /// a path, and set the resume flag. The local file name will be
        /// created or overwritten, but the path must exist.
        /// </summary>
        /// <param name="remFileName"></param>
        /// <param name="locFileName"></param>
        /// <param name="resume"></param>
        public void Download(string remFileName, string locFileName, Boolean resume)
        {
            if (!this.loggedin) this.Login();

            this.BinaryMode = true;

            Debug.Print("Downloading file " + remFileName + " from " + server + "/" + remotePath + "FtpClient");

            if (locFileName.Equals(""))
            {
                locFileName = remFileName;
            }

            FileStream output = null;

            if (!File.Exists(locFileName))
                output = File.Create(locFileName);

            else
                output = new FileStream(locFileName, FileMode.Open);

            Socket cSocket = createDataSocket();

            long offset = 0;

            if (resume)
            {
                offset = output.Length;

                if (offset > 0)
                {
                    this.sendCommand("REST " + offset);
                    if (this.resultCode != 350)
                    {
                        //Server dosnt support resuming
                        offset = 0;
                        Debug.Print("Resuming not supported:" + result.Substring(4) + "FtpClient");
                    }
                    else
                    {
                        Debug.Print("Resuming at offset " + offset + "FtpClient");
                        output.Seek(offset, SeekOrigin.Begin);
                    }
                }
            }

            this.sendCommand("RETR " + remFileName);

            if (this.resultCode != 150 && this.resultCode != 125)
            {
                throw new FtpException(this.result.Substring(4));
            }

            DateTime timeout = DateTime.Now.AddSeconds(this.timeoutSeconds);

            while (timeout > DateTime.Now)
            {
                this.bytes = cSocket.Receive(buffer, buffer.Length, 0);
                output.Write(this.buffer, 0, this.bytes);
                //Thread.Sleep(1);
                if (this.bytes <= 0)
                {
                    break;
                }
            }

            output.Close();

            try { cSocket.Close(); }
            catch (Exception) { }

            this.readResponse();

            if (this.resultCode != 226 && this.resultCode != 250)
                throw new FtpException(this.result.Substring(4));
        }







        /// <summary>
        /// 
        /// </summary>
        private void readResponse()
        {
            this.message = "";
            this.result = this.readLine();

            if (this.result.Length > 3)
                this.resultCode = int.Parse(this.result.Substring(0, 3));
            else
                this.result = null;
        }

        /// <summary>
        /// 
        /// </summary>
        /// <returns></returns>
        private string readLine()
        {
            while (true)
            {
                this.bytes = clientSocket.Receive(this.buffer, this.buffer.Length, 0);
                char[] cc = ASCII.GetChars(this.buffer);
                this.message += new string(cc).Substring(0, this.bytes);
                //this.message += ASCII.GetString(this.buffer, 0, this.bytes);

                if (this.bytes < this.buffer.Length)
                {
                    break;
                }
            }

            string[] msg = this.message.Split('\n');

            if (this.message.Length > 2)
                this.message = msg[msg.Length - 2];

            else
                this.message = msg[0];


            if (this.message.Length > 4 && !this.message.Substring(3, 1).Equals(" ")) return this.readLine();

            if (this.verboseDebugging)
            {
                for (int i = 0; i < msg.Length - 1; i++)
                {
                    Debug.Print(msg[i] + "FtpClient");
                }
            }

            return message;
        }

        /// <summary>
        /// 
        /// </summary>
        /// <param name="command"></param>
        private void sendCommand(String command)
        {
            if (this.verboseDebugging) Debug.Print(command + "FtpClient");

            //Byte[] cmdBytes = Encoding.UTF8.GetBytes(command + "\r\n");

            Byte[] cmdBytes = ASCII.GetBytes(command + "\r\n");
            clientSocket.Send(cmdBytes, cmdBytes.Length, 0);
            this.readResponse();
        }


        /// <summary>
        /// when doing data transfers, we need to open another socket for it.
        /// </summary>
        /// <returns>Connected socket</returns>
        private Socket createDataSocket()
        {
            this.sendCommand("PASV");

            if (this.resultCode != 227) throw new FtpException(this.result.Substring(4));

            int index1 = this.result.IndexOf('(');
            int index2 = this.result.IndexOf(')');

            string ipData = this.result.Substring(index1 + 1, index2 - index1 - 1);

            int[] parts = new int[6];
            string[] sparts = ipData.Split(',');
            for (int i = 0; i < parts.Length; i++)
            {
                parts[i] = int.Parse(sparts[i]);
            }

            int len = ipData.Length;
            int partCount = 0;
            string buf = "";

            //for (int i = 0; i < len && partCount <= 6; i++)
            //{
            //    char[] charr = ipData.Substring(i, 1).ToCharArray();
            //    char ch = charr[0];

            //    if (int.Parse(ch.ToString()) >= 0)
            //        buf += ch;

            //    else if (ch != ',')
            //        throw new FtpException("Malformed PASV result: " + result);

            //    if (ch == ',' || i + 1 == len)
            //    {
            //        try
            //        {
            //            parts[partCount++] = int.Parse(buf);
            //            buf = "";
            //        }
            //        catch (Exception ex)
            //        {
            //            throw new FtpException("Malformed PASV result (not supported?): " + this.result, ex);
            //        }
            //    }
            //}

            string ipAddress = parts[0] + "." + parts[1] + "." + parts[2] + "." + parts[3];

            int port = (parts[4] << 8) + parts[5];

            Socket socket = null;
            IPEndPoint ep = null;

            try
            {
                socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);

                ep = new IPEndPoint(IPAddress.Parse(ipAddress), port);
                socket.Connect(ep);
            }
            catch (Exception ex)
            {
                // doubtfull....
                if (socket != null) socket.Close();

                throw new FtpException("Can't connect to remote server", ex);
            }

            return socket;
        }

        /// <summary>
        /// Always release those sockets.
        /// </summary>
        private void cleanup()
        {
            if (this.clientSocket != null)
            {
                this.clientSocket.Close();
                this.clientSocket = null;
            }
            this.loggedin = false;
        }

        /// <summary>
        /// Destuctor
        /// </summary>
        ~FtpClient()
        {
            this.cleanup();
        }


        /**************************************************************************************************************/
        #region Async methods (auto generated)

        /*
				WinInetApi.FtpClient ftp = new WinInetApi.FtpClient();

				MethodInfo[] methods = ftp.GetType().GetMethods(BindingFlags.DeclaredOnly|BindingFlags.Instance|BindingFlags.Public);

				foreach ( MethodInfo method in methods )
				{
					string param = "";
					string values = "";
					foreach ( ParameterInfo i in  method.GetParameters() )
					{
						param += i.ParameterType.Name + " " + i.Name + ",";
						values += i.Name + ",";
					}
					

					Debug.Print("private delegate " + method.ReturnType.Name + " " + method.Name + "Callback(" + param.TrimEnd(',') + ");");

					Debug.Print("public System.IAsyncResult Begin" + method.Name + "( " + param + " System.AsyncCallback callback )");
					Debug.Print("{");
					Debug.Print("" + method.Name + "Callback ftpCallback = new " + method.Name + "Callback(" + values + " this." + method.Name + ");");
					Debug.Print("return ftpCallback.BeginInvoke(callback, null);");
					Debug.Print("}");
					Debug.Print("public void End" + method.Name + "(System.IAsyncResult asyncResult)");
					Debug.Print("{");
					Debug.Print(method.Name + "Callback fc = (" + method.Name + "Callback) ((AsyncResult)asyncResult).AsyncDelegate;");
					Debug.Print("fc.EndInvoke(asyncResult);");
					Debug.Print("}");
					//Debug.Print(method);
				}
*/


        private delegate void LoginCallback();
        public System.IAsyncResult BeginLogin(System.AsyncCallback callback)
        {
            LoginCallback ftpCallback = new LoginCallback(this.Login);
            return ftpCallback.BeginInvoke(callback, null);
        }
        private delegate void CloseCallback();
        public System.IAsyncResult BeginClose(System.AsyncCallback callback)
        {
            CloseCallback ftpCallback = new CloseCallback(this.Close);
            return ftpCallback.BeginInvoke(callback, null);
        }

        private delegate Int64 GetFileSizeCallback(String fileName);
        public System.IAsyncResult BeginGetFileSize(String fileName, System.AsyncCallback callback)
        {
            GetFileSizeCallback ftpCallback = new GetFileSizeCallback(this.GetFileSize);
            return ftpCallback.BeginInvoke(fileName, callback, null);
        }
        private delegate void DownloadCallback(String remFileName);
        public System.IAsyncResult BeginDownload(String remFileName, System.AsyncCallback callback)
        {
            DownloadCallback ftpCallback = new DownloadCallback(this.Download);
            return ftpCallback.BeginInvoke(remFileName, callback, null);
        }
        private delegate void DownloadFileNameResumeCallback(String remFileName, Boolean resume);
        public System.IAsyncResult BeginDownload(String remFileName, Boolean resume, System.AsyncCallback callback)
        {
            DownloadFileNameResumeCallback ftpCallback = new DownloadFileNameResumeCallback(this.Download);
            return ftpCallback.BeginInvoke(remFileName, resume, callback, null);
        }
        private delegate void DownloadFileNameFileNameCallback(String remFileName, String locFileName);
        public System.IAsyncResult BeginDownload(String remFileName, String locFileName, System.AsyncCallback callback)
        {
            DownloadFileNameFileNameCallback ftpCallback = new DownloadFileNameFileNameCallback(this.Download);
            return ftpCallback.BeginInvoke(remFileName, locFileName, callback, null);
        }
        private delegate void DownloadFileNameFileNameResumeCallback(String remFileName, String locFileName, Boolean resume);
        public System.IAsyncResult BeginDownload(String remFileName, String locFileName, Boolean resume, System.AsyncCallback callback)
        {
            DownloadFileNameFileNameResumeCallback ftpCallback = new DownloadFileNameFileNameResumeCallback(this.Download);
            return ftpCallback.BeginInvoke(remFileName, locFileName, resume, callback, null);
        }





        #endregion
    }
}


and here how i use it ;


ps = new PersistentStorage("SD");
            ps.MountFileSystem();

            byte[] ip = { 192, 168, 1, 210 };
            byte[] subnet = { 255, 255, 255, 0 };
            byte[] gateway = { 192, 168, 1, 1 };
            byte[] mac = { 43, 185, 44, 2, 206, 120 };
            WIZnet_W5100.Enable(SPI.SPI_module.SPI1, (Cpu.Pin)FEZ_Pin.Digital.Di10, (Cpu.Pin)FEZ_Pin.Digital.Di9, true); // WIZnet interface on FEZ Connect
            
            NetworkInterface.EnableStaticIP(ip, subnet, gateway, mac);
            NetworkInterface.EnableStaticDns(new byte[] { 192, 168,1, 1 });
            //Dhcp.EnableDhcp(mac);
            Debug.Print("Network settings:");
            Debug.Print("IP Address: " + new IPAddress(NetworkInterface.IPAddress).ToString());
            Debug.Print("Subnet Mask: " + new IPAddress(NetworkInterface.SubnetMask).ToString());
            Debug.Print("Default Getway: " + new IPAddress(NetworkInterface.GatewayAddress).ToString());
            Debug.Print("DNS Server: " + new IPAddress(NetworkInterface.DnsServer).ToString());


FtpClient ftp = new FtpClient("192.168.1.15", "test", "test");
            //ftp.BinaryMode = true;

            ftp.Download("test.MP3", "\\SD\\aa.mp3", true);

it works perfect… but the download speed is 8kb/s
is there any configuration option to change the speed of spi and/or ethernet shield’s speed init functions?


#2

The SPI clock is set the the maximum possible. But 8kbps does not sound correct to me. Is it Kbit or KByte per second?


#3

hi Joe,
please just try the code… you have the devices…
i just put one ethernet shield and one fez domino…

there are 10 pc in office, and i am using local lan, if i use external link , the speed becomes 4.5kb per second…
also i have netduino plus, it actually works 100mbit. i tried the same code there in same local lan, it downloads files at 400-600kb/s…
we started to create our own pcb design using your chips, also we use wiznet as ethernet chip…
netduino uses atmel microprocessor and the ethernet chip is different…
if we move our design to that , we will loose 3-4 weeks…
so i want to know how can i solve download speed problem…
maybe the ethernet shield i have not working properly… i dont know, someone please test the same code and tell me the result…
also i tried the webserver code in wiki, i downloaded and uploaded files to/from device… speed is the same…
so it can not be my code…

i need serious and urgent help…


#4

you are asking the question incorrectly then. If you want consulting assistance you should contact GHI directly from their website.

(SPI is SPI, what you have is all you’re going to get I feel)


#5

Here is a speed analysis performed by FEZ Hero member Mike:

http://www.tinyclr.com/forum/12/1070/
The speeed he got with W5100 is 14.5KBytes per second. which is 116Kbps.
Also we imporved the speed after that test (firmware version 4.1.4.0 and newer) it should be 3 times as fast.

With the file system and code delay overhead I would expect that you get about 8KByte per second which is about 64kbps (worse case). the number should be better.

My advice is that you double check your code and see where the bottle neck is and try to optimize it. and try to send as big as 1000 byte packets in every transaction to overcome the over head that comes with every TCP packet.


#6

For benchmarking purpose , try this this code out that dumps all the received data:

while (timeout > DateTime.Now)
            {
                this.bytes = cSocket.Receive(buffer, buffer.Length, 0);
                //output.Write(this.buffer, 0, this.bytes);
                //Thread.Sleep(1);
                if (this.bytes <= 0)
                {
                    break;
                }
            }

What is the speed now?


#7

Hi,
The createDataSocket source has been truncated in your post. Can you please resubmit?
Thanks


#8

post is over a year old, I doubt you’re going to get anyone putting that back up. Better would be to start a new thread and ask your own questions. It would also help if you updated your profile so we know who we’re speaking to :slight_smile: Welcome to the forum !


#9

No problem, I already fixed it myself. Actually I have merged 2 previous mails so now have a complete ftp client with both upload and download. I shall be posting it a.s.a.p.


#10

nice work ! :wink: