Networkstream does not recieve all data

Archive
1

Hi.

This question is based on a previous question: https://www.ghielectronics.com/community/forum/topic?id=12569

I’m having trouble downloading a file from a .net web Api onto a G120. The file is around 400 kb in size and I have never experienced downloading the file via the Api when connecting through, for instance WebFetch or when running the code below in the full framework.

On the G120, sometimes it works, sometimes not. When not working it is as if the data are not coming to the device. That is, input.DataAvailable is false and remains false. Note that it never fails directly but often after perhaps 1/10 of the data has been downloaded.

Does anyone have any idea of why it fails and have any suggeestion on how I can solve the problem?


public bool GetServerStream(string url, Stream outputStream, HashAlgorithmType checksumHashType, out byte[] checkSum)
		{
			checkSum = null;
			bool succeeded = false;
			byte[] buffer = new byte[4096];
			byte[] oldBuffer = new byte[4096];
			int bytesRead, oldBytesRead, awaitTime;
			long totalBytesRead;
			byte[] check = new byte[0];
			HashAlgorithm hash = new HashAlgorithm(checksumHashType);
			hash.Initialize();

			if (_ethernet.EthernetIsConnected)
			{
				using (HttpByteStreamRequest request = HttpByteStreamRequest.Create(url))
				{
					request.Method = "GET";
					var response = request.GetResponse();

					if (response.StatusCode == 200)
					{
						NetworkStream input = response.GetResponseStream();

						bytesRead = input.Read(buffer, 0, buffer.Length);
						totalBytesRead = bytesRead;

						do
						{
							oldBytesRead = bytesRead;
							oldBuffer = buffer;

							buffer = new byte[4096];

							if (totalBytesRead < response.ContentLength)
							{
								awaitTime = 0;
								while (!input.DataAvailable)
								{
									Thread.Sleep(1000);
									Debug.Print("Awaiting data");
									awaitTime += 1000;

									if (awaitTime >= 1000 * 20)
									{
										throw new HttpGetException("Server stream timeout");
									}
								}


								bytesRead = input.Read(buffer, 0, buffer.Length);
								totalBytesRead += bytesRead;
								Thread.Sleep(500);
								Debug.Print("Read: " + bytesRead.ToString());
							}
							else
							{
								bytesRead = 0;
							}

							if (bytesRead == 0)
							{
								hash.TransformFinalBlock(oldBuffer, 0, oldBytesRead);
							}
							else
							{
								hash.TransformBlock(oldBuffer, 0, oldBytesRead, oldBuffer, 0);
							}

							outputStream.Write(oldBuffer, 0, oldBytesRead);

						}
						while (bytesRead > 0);

						checkSum = hash.Hash;
						succeeded = true;
						input.Close();
					}
					else
					{
						succeeded = false;
					}
				}
			}
			else
			{
				throw new HttpGetException("Ethernet is down");
			}

			return succeeded;
		}
2

Hi again and thank’s for the reply.

The problem is that after recieving a certain amount of bytes, the socket.Available always return < 1. It does not matter how long I wait, no more bytes becomes available. So for instance, if I call your method GetResponse(…) I still only get a part of the total data, and if I call the method again, socket.Available will return < 1 indefinitely.

I’m connecting the device to internet via a router. Can this affect somehow? (Some preliminary tests suggests that it works better when connecting direclty to internet without a router).

3

Ok… I think I have identified the problem. It is the router, or at least I do not experience any problems when connecting the device directly to internet.

I still need to wait for data to arrive at the socket buffer sometimes but without the router, the data eventually comes.

The file we are trying to download is a software update for the device and since the device is going to be used by our customers behind their router we have a problem because we will not be able to update their software remotely.

Does anyone have any idea of why the router affects the byte transfer?

4

The problem is not specific to a router but more general. I have tried a router from another brand and get the exact same problem. Almost always Socket.Available < 1 after reading 32880 bytes. I have tried switching of NAT in the router without effect.

I don’t know if it matters but I’m using the ENC28J60 module together with the drivers provided by the premium library.

The only solution I can think of is to send the file as partial content.

5

Or it could be a combination. As I said before, I do not have any problems downloading the file using the standard .net framework.

Perhaps the transfer takes too long time for the router to keep the connection open… my knowledge of networking is very limited but it seems as if the router does not know that to do with the subsequent incoming bytes. (I have monitored the tcp communication on the server with wireshark and it tries to send packages but get [TCP Retransmission] [Tcp segment of a reassembled pdu]).

6

Do you have the same issue when downloading the file via raw socket communications, or does it only effect the Web API’s?

7

I have this same issue using raw sockets.

8

@ andre.m
Hmm… this is really weird but your suggestion actually works. So far it has succeeded to read the stream 100% of the times. I have tried it using different routers (Netgear and Dovado).

The following code works perfectly when I’m not behind a router:


while (true)
{

       while (response.DataAvailable < 1)
       {
             Thread.Sleep(200);
               Debug.Print("Waiting for data");
         }

          bytesRead = response.Read(buffer, 0, buffer.Length);
          totalBytesRead += bytesRead;
          Debug.Print("Bytes read: " + bytesRead + " Total bytes read: " + totalBytesRead);
 }

But if I connect to internet via a router, response.DataAvailable will often return < 1 forever when around 32000 bytes have been read.

If I instead use the code you suggested:


while (true)
{
	if (response.DataAvailable > 0)
	{ 
		bytesRead = response.Read(buffer, 0, buffer.Length);
		totalBytesRead += bytesRead;
		Debug.Print("Bytes read: " + bytesRead + " Total bytes read: " + totalBytesRead);
	}
}

it works perfectly (but a bit slow).

The consequence of this is that if your are using a router, then the easy way of reading a stream via HttpResponse.GetResponseStream() won’t work.


Stream s = response.GetResponseStream();
while((bytesRead = s.Read(buffer, 0, buffer.length)) > 0)
{
    totalBytesRead += bytesRead;
}

Stream.Read(…) will hang.

But if I change the code into


while (true)
{
     if (responseStream.Length > 0)
     {
         bytesRead = responseStream.Read(buffer, 0, buffer.Length);
         totalBytesRead += bytesRead;
         Debug.Print("BytesRead: " + bytesRead + " TotalBytesRead: " + totalBytesRead);
       }
}

it works perfectly.

I would love to see if anyone can explain why this is happening and why Andre’s suggestion works. Thank you Andre!