HttpWebRequest - Problem

_Dejan · November 27, 2011, 8:53am

I using DownloadFile() method from Pyxis2 to download HEX files from HTTP server but something do not work ok.

Here is code:


        public static int DownloadFile(string url, string file)
        {
            return DownloadFile(url, file);
        }

        public static int DownloadFile(string url, string file, int bufferSize = 1024)
        {
            try
            {
                // Ensure Directory
                if (!Directory.Exists(Path.GetDirectoryName(file)))
                    Directory.CreateDirectory(Path.GetDirectoryName(file));

                // Perpare for file
                if (File.Exists(file))
                {
                    File.Delete(file);
                    VolumeInfo sdvolumeinfo = new VolumeInfo(Module.storage);
                    sdvolumeinfo.FlushAll();
                }
                FileStream FS = new FileStream(file, FileMode.CreateNew, FileAccess.Write);

                byte[] b = new byte[bufferSize];

                HttpWebRequest request = HttpWebRequest.Create(url) as HttpWebRequest;
                request.KeepAlive = false;
          
                WebResponse resp = request.GetResponse();

                // Get the network response stream to read the page data.
                if (resp != null)
                {
                    Stream respStream = resp.GetResponseStream();
                    int bytesRead = 0;
                    Decoder UTF8decoder = System.Text.Encoding.UTF8.GetDecoder();
                    int totalRead = 0;

                    // allow 5 seconds for reading the stream
                    respStream.ReadTimeout = 5000;

                    // If we know the content length, read exactly that amount of 
                    // data; otherwise, read until there is nothing left to read.
                    if (resp.ContentLength != -1)
                    {
                        for (int dataRem = (int)resp.ContentLength; dataRem > 0; )
                        {
                            //Thread.Sleep(50);
                            bytesRead = respStream.Read(b, 0, (dataRem > b.Length) ? b.Length : dataRem);
                            FS.Write(b, 0, bytesRead);

                            if (bytesRead == 0)
                                throw new Exception("Data less than expected");

                            totalRead += bytesRead;
                            dataRem -= bytesRead;
                        }

                    }
                    else
                    {
                        // Attempt to read chunks
                        b = new byte[1024];
                        string strOut = string.Empty;

                        while (true)
                        {
                            Thread.Sleep(500);

                            if (b.Length > respStream.Length)
                                b = new byte[respStream.Length];

                            bytesRead = respStream.Read(b, 0, b.Length);
                            if (bytesRead == 0)
                            {
                                break;
                            }
                            else
                                FS.Write(b, 0, bytesRead);

                        }
                    }

                    // Close the response stream.  For Keep-Alive streams, the 
                    // stream will remain open and will be pushed into the unused 
                    // stream list.
                    resp.Close();
                    request.Dispose();

                    FS.Close();
                    VolumeInfo sdvolumeinfo = new VolumeInfo(Module.storage);
                    sdvolumeinfo.FlushAll();
                    return totalRead;
                }

                return 0;
            }
            catch (Exception e)
            {
                return 0;
            }
        }

Module.storage=“SD”;

for example I call this:


Debug.GC(true);
Module.WriteDebugLog("Update.DownloadUpdates() - File: app.hex Size: " + Network.DownloadFile("http://www.domain.com/updates/app.hex", UpdateFolder + "app.hex", 5120));
Debug.GC(true);
Module.WriteDebugLog("Update.DownloadUpdates() - File: bootloader.hex Size: " + Network.DownloadFile("http://www.domain.com/updates/bootloader.hex", UpdateFolder + "bootloader.hex", 5120));
Debug.GC(true);
Module.WriteDebugLog("Update.DownloadUpdates() - File: CLR.HEX Size: " + Network.DownloadFile("http://www.domain.com/updates/CLR.HEX", UpdateFolder + "CLR.HEX", 5120));
Debug.GC(true);
Module.WriteDebugLog("Update.DownloadUpdates() - File: CLR2.HEX Size: " + Network.DownloadFile("http://www.domain.com/updates/CLR2.HEX", UpdateFolder + "CLR2.HEX", 5120));
Debug.GC(true);
Module.WriteDebugLog("Update.DownloadUpdates() - File: Config.HEX Size: " + Network.DownloadFile("http://www.domain.com/updates/Config.HEX", UpdateFolder + "Config.HEX", 5120));
Debug.GC(true);
Module.WriteDebugLog("Update.DownloadUpdates() - Files Downloading Completed");

UpdateFolder = @ “\SD\update”;

Error:


GC: 26msec 278256 bytes used, 9617208 bytes available
Type 0F (STRING              ):   5928 bytes
Type 11 (CLASS               ):  17400 bytes
Type 12 (VALUETYPE           ):    684 bytes
Type 13 (SZARRAY             ):  37848 bytes
Type 15 (FREEBLOCK           ): 9617208 bytes
Type 16 (CACHEDBLOCK         ):   3912 bytes
Type 17 (ASSEMBLY            ):  31836 bytes
Type 18 (WEAKCLASS           ):    144 bytes
Type 19 (REFLECTION          ):    192 bytes
Type 1B (DELEGATE_HEAD       ):   1656 bytes
Type 1C (DELEGATELIST_HEAD   ):    180 bytes
Type 1D (OBJECT_TO_EVENT     ):    696 bytes
Type 1E (BINARY_BLOB_HEAD    ): 167256 bytes
Type 1F (THREAD              ):   2688 bytes
Type 20 (SUBTHREAD           ):    336 bytes
Type 21 (STACK_FRAME         ):   2712 bytes
Type 22 (TIMER_HEAD          ):    144 bytes
Type 27 (FINALIZER_HEAD      ):    816 bytes
Type 31 (IO_PORT             ):    684 bytes
Type 34 (APPDOMAIN_HEAD      ):     72 bytes
Type 36 (APPDOMAIN_ASSEMBLY  ):   3072 bytes
    #### Exception System.Net.Sockets.SocketException - CLR_E_FAIL (16) ####
    #### Message: 
    #### Microsoft.SPOT.Net.SocketNative::recv [IP: 0000] ####
    #### System.Net.Sockets.Socket::Receive [IP: 0018] ####
    #### System.Net.Sockets.NetworkStream::Read [IP: 0062] ####
    #### System.Net.InputNetworkStreamWrapper::ReadInternal [IP: 00d7] ####
    #### System.Net.InputNetworkStreamWrapper::Read [IP: 000d] ####
    #### PEPP.Network::DownloadFile [IP: 00b4] ####
    #### PEPP.Update::DownloadUpdates [IP: 01ac] ####
    #### PEPP.Update::UpdateThread [IP: 0068] ####
    #### SocketException ErrorCode = 10060
    #### SocketException ErrorCode = 10060
A first chance exception of type 'System.Net.Sockets.SocketException' occurred in Microsoft.SPOT.Net.dll
    #### SocketException ErrorCode = 10060
    #### SocketException ErrorCode = 10060
11/01/2011 06:57:01: Update.DownloadUpdates() - File: app.hex Size: 0
GC: 11msec 274092 bytes used, 9621372 bytes available
Type 0F (STRING              ):   6936 bytes
Type 11 (CLASS               ):  18720 bytes
Type 12 (VALUETYPE           ):    684 bytes
Type 13 (SZARRAY             ):  29856 bytes
Type 15 (FREEBLOCK           ): 9621372 bytes
Type 16 (CACHEDBLOCK         ):    240 bytes
Type 17 (ASSEMBLY            ):  31836 bytes
Type 18 (WEAKCLASS           ):    144 bytes
Type 19 (REFLECTION          ):    192 bytes
Type 1B (DELEGATE_HEAD       ):   1656 bytes
Type 1C (DELEGATELIST_HEAD   ):    180 bytes
Type 1D (OBJECT_TO_EVENT     ):    696 bytes
Type 1E (BINARY_BLOB_HEAD    ): 171852 bytes
Type 1F (THREAD              ):   3072 bytes
Type 20 (SUBTHREAD           ):    384 bytes
Type 21 (STACK_FRAME         ):   2712 bytes
Type 22 (TIMER_HEAD          ):    144 bytes
Type 27 (FINALIZER_HEAD      ):    960 bytes
Type 31 (IO_PORT             ):    684 bytes
Type 34 (APPDOMAIN_HEAD      ):     72 bytes
Type 36 (APPDOMAIN_ASSEMBLY  ):   3072 bytes
    #### Exception System.Net.Sockets.SocketException - CLR_E_FAIL (16) ####
    #### Message: 
    #### Microsoft.SPOT.Net.SocketNative::recv [IP: 0000] ####
    #### System.Net.Sockets.Socket::Receive [IP: 0018] ####
    #### System.Net.Sockets.NetworkStream::Read [IP: 0062] ####
    #### System.Net.InputNetworkStreamWrapper::ReadInternal [IP: 00d7] ####
    #### System.Net.InputNetworkStreamWrapper::Read [IP: 000d] ####
    #### PEPP.Network::DownloadFile [IP: 00b4] ####
    #### PEPP.Update::DownloadUpdates [IP: 01ac] ####
    #### PEPP.Update::UpdateThread [IP: 0068] ####
    #### SocketException ErrorCode = 10060
    #### SocketException ErrorCode = 10060
A first chance exception of type 'System.Net.Sockets.SocketException' occurred in Microsoft.SPOT.Net.dll
    #### SocketException ErrorCode = 10060
    #### SocketException ErrorCode = 10060
11/01/2011 06:57:17: Update.DownloadUpdates() - File: bootloader.hex Size: 0
GC: 10msec 291288 bytes used, 9604176 bytes available
Type 0F (STRING              ):   6912 bytes
Type 11 (CLASS               ):  18360 bytes
Type 12 (VALUETYPE           ):    684 bytes
Type 13 (SZARRAY             ):  47604 bytes
Type 15 (FREEBLOCK           ): 9604176 bytes
Type 16 (CACHEDBLOCK         ):     96 bytes
Type 17 (ASSEMBLY            ):  31836 bytes
Type 18 (WEAKCLASS           ):    144 bytes
Type 19 (REFLECTION          ):    192 bytes
Type 1B (DELEGATE_HEAD       ):   1656 bytes
Type 1C (DELEGATELIST_HEAD   ):    180 bytes
Type 1D (OBJECT_TO_EVENT     ):    696 bytes
Type 1E (BINARY_BLOB_HEAD    ): 171852 bytes
Type 1F (THREAD              ):   3072 bytes
Type 20 (SUBTHREAD           ):    384 bytes
Type 21 (STACK_FRAME         ):   2712 bytes
Type 22 (TIMER_HEAD          ):    144 bytes
Type 27 (FINALIZER_HEAD      ):    936 bytes
Type 31 (IO_PORT             ):    684 bytes
Type 34 (APPDOMAIN_HEAD      ):     72 bytes
Type 36 (APPDOMAIN_ASSEMBLY  ):   3072 bytes
    #### Exception System.Net.Sockets.SocketException - CLR_E_FAIL (16) ####
    #### Message: 
    #### Microsoft.SPOT.Net.SocketNative::recv [IP: 0000] ####
    #### System.Net.Sockets.Socket::Receive [IP: 0018] ####
    #### System.Net.Sockets.NetworkStream::Read [IP: 0062] ####
    #### System.Net.InputNetworkStreamWrapper::ReadInternal [IP: 00d7] ####
    #### System.Net.InputNetworkStreamWrapper::Read [IP: 000d] ####
    #### PEPP.Network::DownloadFile [IP: 00b4] ####
    #### PEPP.Update::DownloadUpdates [IP: 01ac] ####
    #### PEPP.Update::UpdateThread [IP: 0068] ####
    #### SocketException ErrorCode = 10060
    #### SocketException ErrorCode = 10060
A first chance exception of type 'System.Net.Sockets.SocketException' occurred in Microsoft.SPOT.Net.dll
    #### SocketException ErrorCode = 10060
    #### SocketException ErrorCode = 10060
11/01/2011 06:57:29: Update.DownloadUpdates() - File: CLR.HEX Size: 0
GC: 10msec 450564 bytes used, 9444900 bytes available
Type 0F (STRING              ):   6864 bytes
Type 11 (CLASS               ):  18384 bytes
Type 12 (VALUETYPE           ):    684 bytes
Type 13 (SZARRAY             ):  53412 bytes
Type 15 (FREEBLOCK           ): 9444900 bytes
Type 16 (CACHEDBLOCK         ):     96 bytes
Type 17 (ASSEMBLY            ):  31836 bytes
Type 18 (WEAKCLASS           ):    144 bytes
Type 19 (REFLECTION          ):    192 bytes
Type 1B (DELEGATE_HEAD       ):   1656 bytes
Type 1C (DELEGATELIST_HEAD   ):    180 bytes
Type 1D (OBJECT_TO_EVENT     ):    696 bytes
Type 1E (BINARY_BLOB_HEAD    ): 325548 bytes
Type 1F (THREAD              ):   3072 bytes
Type 20 (SUBTHREAD           ):    384 bytes
Type 21 (STACK_FRAME         ):   2508 bytes
Type 22 (TIMER_HEAD          ):    144 bytes
Type 27 (FINALIZER_HEAD      ):    936 bytes
Type 31 (IO_PORT             ):    684 bytes
Type 34 (APPDOMAIN_HEAD      ):     72 bytes
Type 36 (APPDOMAIN_ASSEMBLY  ):   3072 bytes
    #### Exception System.Net.Sockets.SocketException - CLR_E_FAIL (16) ####
    #### Message: 
    #### Microsoft.SPOT.Net.SocketNative::recv [IP: 0000] ####
    #### System.Net.Sockets.Socket::Receive [IP: 0018] ####
    #### System.Net.Sockets.NetworkStream::Read [IP: 0062] ####
    #### System.Net.InputNetworkStreamWrapper::ReadInternal [IP: 00d7] ####
    #### System.Net.InputNetworkStreamWrapper::Read [IP: 000d] ####
    #### PEPP.Network::DownloadFile [IP: 00b4] ####
    #### PEPP.Update::DownloadUpdates [IP: 01ac] ####
    #### PEPP.Update::UpdateThread [IP: 0068] ####
    #### SocketException ErrorCode = 10060
    #### SocketException ErrorCode = 10060
A first chance exception of type 'System.Net.Sockets.SocketException' occurred in Microsoft.SPOT.Net.dll
    #### SocketException ErrorCode = 10060
    #### SocketException ErrorCode = 10060
11/01/2011 06:57:44: Update.DownloadUpdates() - File: CLR2.HEX Size: 0
GC: 10msec 436716 bytes used, 9458748 bytes available
Type 0F (STRING              ):   6912 bytes
Type 11 (CLASS               ):  18384 bytes
Type 12 (VALUETYPE           ):    684 bytes
Type 13 (SZARRAY             ):  39516 bytes
Type 15 (FREEBLOCK           ): 9458748 bytes
Type 16 (CACHEDBLOCK         ):     96 bytes
Type 17 (ASSEMBLY            ):  31836 bytes
Type 18 (WEAKCLASS           ):    144 bytes
Type 19 (REFLECTION          ):    192 bytes
Type 1B (DELEGATE_HEAD       ):   1656 bytes
Type 1C (DELEGATELIST_HEAD   ):    180 bytes
Type 1D (OBJECT_TO_EVENT     ):    696 bytes
Type 1E (BINARY_BLOB_HEAD    ): 325548 bytes
Type 1F (THREAD              ):   3072 bytes
Type 20 (SUBTHREAD           ):    384 bytes
Type 21 (STACK_FRAME         ):   2508 bytes
Type 22 (TIMER_HEAD          ):    144 bytes
Type 27 (FINALIZER_HEAD      ):    936 bytes
Type 31 (IO_PORT             ):    684 bytes
Type 34 (APPDOMAIN_HEAD      ):     72 bytes
Type 36 (APPDOMAIN_ASSEMBLY  ):   3072 bytes
    #### Exception System.Net.Sockets.SocketException - CLR_E_FAIL (16) ####
    #### Message: 
    #### Microsoft.SPOT.Net.SocketNative::recv [IP: 0000] ####
    #### System.Net.Sockets.Socket::Receive [IP: 0018] ####
    #### System.Net.Sockets.NetworkStream::Read [IP: 0062] ####
    #### System.Net.InputNetworkStreamWrapper::ReadInternal [IP: 00d7] ####
    #### System.Net.InputNetworkStreamWrapper::Read [IP: 000d] ####
    #### PEPP.Network::DownloadFile [IP: 00b4] ####
    #### PEPP.Update::DownloadUpdates [IP: 01ac] ####
    #### PEPP.Update::UpdateThread [IP: 0068] ####
    #### SocketException ErrorCode = 10060
    #### SocketException ErrorCode = 10060
A first chance exception of type 'System.Net.Sockets.SocketException' occurred in Microsoft.SPOT.Net.dll
    #### SocketException ErrorCode = 10060
    #### SocketException ErrorCode = 10060
11/01/2011 06:57:58: Update.DownloadUpdates() - File: Config.HEX Size: 0
GC: 10msec 282852 bytes used, 9612612 bytes available
Type 0F (STRING              ):   6948 bytes
Type 11 (CLASS               ):  18432 bytes
Type 12 (VALUETYPE           ):    684 bytes
Type 13 (SZARRAY             ):  37848 bytes
Type 15 (FREEBLOCK           ): 9612612 bytes
Type 16 (CACHEDBLOCK         ):     96 bytes
Type 17 (ASSEMBLY            ):  31836 bytes
Type 18 (WEAKCLASS           ):    144 bytes
Type 19 (REFLECTION          ):    192 bytes
Type 1B (DELEGATE_HEAD       ):   1656 bytes
Type 1C (DELEGATELIST_HEAD   ):    180 bytes
Type 1D (OBJECT_TO_EVENT     ):    696 bytes
Type 1E (BINARY_BLOB_HEAD    ): 172956 bytes
Type 1F (THREAD              ):   3072 bytes
Type 20 (SUBTHREAD           ):    384 bytes
Type 21 (STACK_FRAME         ):   2712 bytes
Type 22 (TIMER_HEAD          ):    144 bytes
Type 23 (LOCK_HEAD           ):     60 bytes
Type 24 (LOCK_OWNER_HEAD     ):     24 bytes
Type 27 (FINALIZER_HEAD      ):    960 bytes
Type 31 (IO_PORT             ):    684 bytes
Type 34 (APPDOMAIN_HEAD      ):     72 bytes
Type 36 (APPDOMAIN_ASSEMBLY  ):   3072 bytes
11/01/2011 06:57:59: Update.DownloadUpdates() - Files Downloading Completed
The thread '<No Name>' (0x10) has exited with code 0 (0x0).

Any idea how to fix it? Im try add:
request.Timeout = 20000;
but do not help…

_Dejan · November 27, 2011, 9:07am

If I check files on SD card:
app.hex 117KB (must be around 2500KB)
bootloader.hex 3KB (must be around 200KB)
CLR.HEX 2KB (must be around 1500KB)
CLR2.HEX 8KB (must be around 1500KB)
Config.HEX 7KB (must be around 7KB)

godFather89 · November 28, 2011, 3:45am

The else should be treated like this

byte[] b = new byte[1024];
// Read until the end of the data is reached.
while (true)
{
    // If the Read method times out, it throws an exception, 
    // which is expected for Keep-Alive streams because the 
    // connection isn't terminated.
    try
    {
        Thread.Sleep(500);
        bytesRead =
            respStream.Read(b, 0, b.Length);
    }
    catch (Exception)
    {
        bytesRead = 0;
    }

    // Zero bytes indicates the connection has been closed 
    // by the server.
    if (bytesRead == 0)
    {
        break;
    }

    FS.Write(b, 0, bytesRead);
}

_Dejan · November 28, 2011, 5:47am

Hi godFather89,
Code is copied from Pyxis2 source and Im see that a lot of users on forum recommend to use it…
What Im find until now:
-Pyxis2 DownloadFile() Work OK when I download file by Ethernet connection.
-Pyxis2 DownloadFile() Do NOT Work ok when I download file by GPRS(PPP) connection.

Also it use HttpWebRequest&WebResponse which is VERY SLOW. simple 100-500bytes request take few seconds(2-5) to complete on PC need 100-200ms to get response from server…

Im make own WebClient class to replace Pyxis2 Download functions and use some parts of GoogleMaps example to sucessfully make Ethernet and PPP download work as must.

godFather89 · November 28, 2011, 6:40am

Try to remove or decrease the sleep time. (in Thread.Sleep(500)
Maybe this would speed up things a little

LE: I noticed now that the request takes too long. Is this on the PPP connection? And if you’re using the PPP modem on the PC?