Problems with buffer size

[WildBamaBoy]

In my client/server program, whenever I need to get data from one or the other it requires a buffer to temporarily store the data. The problem is that I don't know how much data I am going to be getting. I could be receiving a few lines of text or a 100mb file.

I've tried:

1. Getting the Length of the NetworkStream.
Not supported. It would be great if it worked! :'(

2. Reading NetworkStream with a StreamReader and getting the length
The length stops at 8 kilobytes.

3. Increasing the size of the buffer as needed
Data became corrupted.

4. Sending the size of the data and then the actual data
The client gets one thing from the server and doesn't receive anything after that.

I'm completely out of ideas now. Can anyone help me with this?

Here's the sending and receiving code for my server with the attempt and number 4 included.

public static object Receive()
        {
            MemoryStream MS;
            BinaryFormatter BF = new BinaryFormatter();

            //Receive the size of the data.
            int DataSize = 0;
            byte[] SizeBuffer = new byte[1024];
            int k = Sock.Receive(SizeBuffer);

            DataSize = BitConverter.ToInt32(SizeBuffer, 0);

            // Set the size of the buffer and get the actual data
            List<byte> ReceivedBytes = new List<byte>();
            byte[] DataBuffer = new byte[DataSize];

            k = Sock.Receive(DataBuffer);

            for (int i = 0; i < k; i++)
            {
                ReceivedBytes.Add(DataBuffer[i]);
            }

            MS = new MemoryStream(ReceivedBytes.ToArray());

            return BF.Deserialize(MS);
        }

        public static void Send(object Data)
        {
            MemoryStream MS = new MemoryStream();
            BinaryFormatter BF = new BinaryFormatter();

            BF.Serialize(MS, Data);

            Sock.Send(BitConverter.GetBytes(MS.ToArray().Length)); //send the length
            Sock.Send(MS.ToArray()); // and then the data
        }

And this is the send and receive code for the client. Almost identical the server code.

public object Receive()
        {
            MemoryStream MS;
            BinaryFormatter BF = new BinaryFormatter();

            //Receive the size of the data.
            int DataSize = 0;
            byte[] SizeBuffer = new byte[1024];
            int k = NetStream.Read(SizeBuffer, 0, SizeBuffer.Length);

            DataSize = BitConverter.ToInt32(SizeBuffer, 0);

            // Set the size of the buffer and get the actual data
            List<byte> ReceivedBytes = new List<byte>();
            byte[] DataBuffer = new byte[DataSize];

            k = NetStream.Read(DataBuffer, 0, DataBuffer.Length);

            for (int i = 0; i < k; i++)
            {
                ReceivedBytes.Add(DataBuffer[i]);
            }

            MS = new MemoryStream(ReceivedBytes.ToArray());

            return BF.Deserialize(MS);
        }

        public void Send(object Data)
        {
            MemoryStream MS = new MemoryStream();
            BinaryFormatter BF = new BinaryFormatter();

            BF.Serialize(MS, Data);

            byte[] LengthOfDataAsBytes = BitConverter.GetBytes(MS.ToArray().Length);
            NetStream.Write(LengthOfDataAsBytes, 0, LengthOfDataAsBytes.Length); //Write the length
            NetStream.Write(MS.ToArray(), 0, MS.ToArray().Length); //and then the data
        }

Absolutely any help is appreciated.

[jfarrugia]

Couple questions:

a) in the above implementation, is any exception being triggered somewhere or is data being lost?

b) i always use that method.. creating a header prior to sending payload which would include the length of the payload.. normally i set the first 4 to 6 bytes manually as a payload length and read the first 4 or 6 bytes when receiving header

Technically, the principle you're using is correct

try creating a byte[] and copy the length + payload in one byte[] on server and write to stream all at once...

[WildBamaBoy]

A. It's really strange...sometimes the sending and receiving works once, and sometimes not at all. An exception is never thrown, the client just seems to indefinitely be waiting for data when the server has already sent it.

B. That sounds like it would work well....I'll give it a shot.

[WildBamaBoy]

Gah. It's so weird! I did what you suggested. I can send a password to the server and get a response, but when trying to get bigger amounts of data everything goes crazy. The data I am sending to the client is 6306 bytes exactly. If I step through all of the code line by line it works great. However if I run the program normally the client throws a Serialization Exception.

The exception says "End of Stream encountered before parsing was completed."

EDIT: Putting a breakpoint on the moment when my client deserializes the data showed that it only received 4372 of the 6306 bytes it was supposed to be getting. The server shows that it sends all of it, too. :'( Why would I not be receiving some data when I don't step through it line by line?

EDIT2: This is the new code for the server.

public static object Receive()
        {
            MemoryStream MS;
            BinaryFormatter BF = new BinaryFormatter();

            //Receive the size of the data.
            List<byte> ReceivedBytes = new List<byte>();

            byte[] Buffer = new byte[1024];
            int BytesRecieved = Sock.Receive(Buffer);

            for (int i = 0; i < BytesRecieved; i++)
            {
                ReceivedBytes.Add(Buffer[i]);
            }

            MS = new MemoryStream(ReceivedBytes.ToArray());

            return BF.Deserialize(MS);
        }

        public static void Send(object Data)
        {
            MemoryStream MS = new MemoryStream();
            BinaryFormatter BF = new BinaryFormatter();

            BF.Serialize(MS, Data);

            byte[] Payload = new byte[8 + MS.ToArray().Length];
            byte[] ByteData = MS.ToArray();
            byte[] LengthOfData = BitConverter.GetBytes(ByteData.Length);

            System.Buffer.BlockCopy(LengthOfData, 0, Payload, 0, LengthOfData.Length);
            System.Buffer.BlockCopy(ByteData, 0, Payload, 8, ByteData.Length);

            Sock.Send(Payload);
        }

and the client

public object Receive()
        {
            MemoryStream MS;
            BinaryFormatter BF = new BinaryFormatter();

            //Receive the size of the data.
            List<byte> ReceivedBytes = new List<byte>();

            byte[] Buffer = new byte[8];                        //First 8 bytes are the size of the data
            NetStream.Read(Buffer, 0, 8);                       //Read them
            Buffer = new byte[BitConverter.ToInt32(Buffer, 0)]; //Set the buffer accordingly

            int BytesRecieved = NetStream.Read(Buffer, 0, Buffer.Length);

            for (int i = 0; i < BytesRecieved; i++)
            {
                ReceivedBytes.Add(Buffer[i]);
            }

            MS = new MemoryStream(ReceivedBytes.ToArray());

            return BF.Deserialize(MS);
        }

        public void Send(object Data)
        {
            MemoryStream MS = new MemoryStream();
            BinaryFormatter BF = new BinaryFormatter();

            BF.Serialize(MS, Data);

            NetStream.Write(MS.ToArray(), 0, MS.ToArray().Length);
        }

[WildBamaBoy]

I tried with the client and server running on two different computers and it actually started working. I increased the amount of data sent until it finally crashed. It crashed when the size of the data was 24110 bytes, receiving only 8752.

EDIT: It only works SOMETIMES. :?:

[WildBamaBoy]

Uhh apparently it works now. :S
I actually read what the code was doing and noticed there was garbage left over from how the program used to work. Removed that and it appears to run fine.

[Mitja Bonca]

Hi, would you mind sharing the solution?
thx in advance.

[WildBamaBoy]

Server code

public static object Receive()
        {
            MemoryStream MS;

            byte[] Buffer = new byte[1024];
            int BytesRecieved = Sock.Receive(Buffer);

            MS = new MemoryStream(Buffer);

            return new BinaryFormatter().Deserialize(MS);
        }

        public static void Send(object Data)
        {
            MemoryStream MS = new MemoryStream();
            BinaryFormatter BF = new BinaryFormatter();

            BF.Serialize(MS, Data);

            byte[] Payload = new byte[8 + MS.ToArray().Length];
            byte[] ByteData = MS.ToArray();
            byte[] LengthOfData = BitConverter.GetBytes(ByteData.Length);

            Buffer.BlockCopy(LengthOfData, 0, Payload, 0, LengthOfData.Length);
            Buffer.BlockCopy(ByteData, 0, Payload, 8, ByteData.Length);

            Sock.Send(Payload);

            Log("Sent payload of " + (Payload.Length - 8) + " bytes.");
        }

Client code

public object Receive()
        {
            DataReceivedLabel.Text = "0";
            DataSizeLabel.Text = "0 bytes";

            MemoryStream MS;
            BinaryFormatter BF = new BinaryFormatter();

            byte[] Buffer = new byte[8];                        //First 8 bytes are the size of the data
            NetStream.Read(Buffer, 0, 8);                       //Read them
            Buffer = new byte[BitConverter.ToInt32(Buffer, 0)]; //Set the buffer accordingly

            DataSizeLabel.Text = Buffer.Length + " bytes";

            int BytesRecieved = 0;

            while (BytesRecieved != Buffer.Length)
            {
                try
                {
                    BytesRecieved = BytesRecieved + NetStream.Read(Buffer, BytesRecieved, 100); //Read 100 bytes to the buffer at a time
                    DataReceivedLabel.Text = BytesRecieved.ToString();                          //Set the amount received
                }

                catch (ArgumentOutOfRangeException)                                             //There's less than 100 bytes left to read
                {
                    int Remainder = Buffer.Length - Convert.ToInt32(DataReceivedLabel.Text);    //Get the bytes left to read and finish.

                    BytesRecieved = BytesRecieved + NetStream.Read(Buffer, BytesRecieved, Remainder);
                    DataReceivedLabel.Text = BytesRecieved.ToString();
                }
            }

            MS = new MemoryStream(Buffer);
            return BF.Deserialize(MS);
        }

        public void Send(object Data)
        {
            MemoryStream MS = new MemoryStream();
            BinaryFormatter BF = new BinaryFormatter();

            BF.Serialize(MS, Data);

            NetStream.Write(MS.ToArray(), 0, MS.ToArray().Length);
        }

Compare to above. The 'garbage' was the adding the bytes that were already in the buffer to a List and converting it to a byte array. That used to be a char list and each byte was converted to a char and placed in that list. There was no need to have that List anymore when the buffer already held all of the bytes. Still not sure exactly what was going on but taking that out fixed it. Kinda dumb to not take it out in the first place...hehe :$