TCP transfer not finishing: "An existing connection was forcibly closed by the remote host"?

TCP transfer not finishing: "An existing connection was forcibly closed by
the remote host"?

I have a client application that uses a TcpClient to send a file to a
server application that uses a TcpListener to receive it. Sometimes, the
file transfers fine. But at other times, the transfer starts but does not
finish. After I have read in a varying number of bytes on the server, I
get an IOException with the message: "An existing connection was forcibly
closed by the remote host."
On the client side, I create a header byte array containing data about my
file, including the total size in bytes and a few other bits of data. I
combine this byte array with that of the file, and send it to the server.
TcpClient fileClient = new TcpClient();
fileClient.Connect("mydomain.com", 7728);
NetworkStream clientStream = fileClient.GetStream();
ASCIIEncoding encoder = new ASCIIEncoding();
FileStream fs = File.Open(filePath, FileMode.Open);
string header = filePath + "|" + fs.Length.ToString() + "|" +
this.ID); //ID = short string of mine
header = header.PadRight(512, '*');
byte[] sizeArray = encoder.GetBytes(header);
byte[] fileBuffer = new byte[fs.Length];
fs.Read(fileBuffer, 0, Convert.ToInt32(fs.Length));
fs.Close();
byte[] buffer = Combine(sizeArray, fileBuffer); //Combine = method
I use to combine byte arrays
clientStream.Write(buffer, 0, buffer.Length);
clientStream.Flush();
On the server side, I create a TcpListener, and start listening. When I
get a connection, I handle it, read my header array, the first 512 bytes,
determine the size of the new file, and then use this code to read the
file / rest of the bytes:
int newSize = ... //The size that I sent in my header array
byte[] fileArray = new byte[newSize];
int off = 0;
while (true)
{
try
{
off += clientStream.Read(fileArray, off, fileArray.Length
- off);
}
catch (Exception ex)
{
off = newSize; // Enables partial receive on error, not
total loss
}
if (off >= newSize)
{
break;
}
}
This is where it sometimes - about 25% of the time - goes wrong. I will
get the exception at the off += clientStream.Read(fileArray, off,
fileArray.Length - off) line. I surrounded it with a try catch, which
enables the application to still get part of the file even when this error
occurs. However, I need to be able to get the full file all the time. What
am I doing wrong with this code?
The files I am transferring are JPEG images around 100KB in size, and
generally take no more than two seconds to transfer. Changing the timeout
values for the clients and the listener does nothing to help.