Sending file through socket - Memory Issues

[WildBamaBoy]

Here's my dilema. My client program is going to request a certain file from my server program, so the server program needs to send the size of the file it requests and then the file itself by using a Socket all in one go. My client will then read the size of the data, read the file to a buffer, and write the buffer to a file. Essentially it's just downloading something from the computer my server program is running on.

As the files get larger I'm running in to memory issues. Here's how I send something from the server, with an old attempt at using FileStreams to use less memory. I was using MemoryStreams. Send(File.ReadAllBytes(@"C:\largevideo.avi"));

public void Send(object Data)
        {
            FileStream SerializeStream = new FileStream(Properties.Settings.Default.TempPath + "serialstrm.TEMP", FileMode.Create); //Create a file to hold serialized data
            BinaryFormatter BF = new BinaryFormatter();

            BF.Serialize(SerializeStream, Data); //serialize the data to the file

            //first 8 bytes = size of data
            byte[] Payload      = new byte[8 + SerializeStream.Length]; 
            
            //stupidity right here :P
            byte[] ByteData     = Data as byte[]; 
            
            //length of data in byte format 
            byte[] LengthOfData = BitConverter.GetBytes(ByteData.Length); 
                                
            //copy length of data to Payload starting at offset 0
            Buffer.BlockCopy(LengthOfData, 0, Payload, 0, LengthOfData.Length); 
            //copy the byte data to Payload starting at offset 8
            Buffer.BlockCopy(ByteData, 0, Payload, 8, ByteData.Length); 
            

            MasterSocket.Send(Payload);
        }

I can't figure out how I can send this to the client without reading the entire file into memory first. I know that using File.ReadAllBytes() is going to put that into the memory. What if the server or client computer doesn't have that much memory to spare? Can I do this without filling up the memory?

[Momerath]

You are going to have to break it up into chunks and send them that way, just like FTP does. Default buffer size is 4k, so I'd use 4k chunks.

[sknake]

Default buffer sizes may be 4k however the upper limit of data that can be sent depends on PMTU (Path MTU) between the two endpoints. Typically this will be in the area of 1500 bytes (less the bytes used for protocol headers)

Reference a similar thread sending files over TCP for examples.

To stream a file:

void SendFile()
    {
      using (FileStream fs = new FileStream(@"C:\...\file.txt", FileMode.Open))
      {
        byte[] buffer = new byte[1024];
        int len;
        while ((len = fs.Read(buffer, 0, buffer.Length)) > 0)
        {
          //Socket.Send(buffer, 0, len);
        }
      }
    }

[WildBamaBoy]

I've been wondering this for awhile... On your line 3 why should I use "using" before making the FileStream?

I haven't attempted yet, but just wondering. ;)

[sknake]

The using(){} clause ensures IDisposable objects are disposed and takes the object out of scope when disposed.

MSDN:
http://msdn.microsoft.com/en-us/library/yh598w02.aspx

Example with using:

using (Font font1 = new Font("Arial", 10.0f)) 
{
    byte charset = font1.GdiCharSet;
}

Functionally identical:

{
  Font font1 = new Font("Arial", 10.0f);
  try
  {
    byte charset = font1.GdiCharSet;
  }
  finally
  {
    if (font1 != null)
      ((IDisposable)font1).Dispose();
  }
}

[skatamatic]

I wrote a fragmented file class to handle this sort of thing. This is a simplified version of it:

using System;
using System.Collections.Generic;
using System.Text;
using System.Net.Sockets;
using System.IO;

namespace fragfile
{
    public class FragmentedFileSender
    {
        private string _sourcePath;
        private int _fileBytes;
        private int _PieceSize;

        public FragmentedFileSender(string filePath, int bytesPerPiece)
        {
            _PieceSize = bytesPerPiece;
            _sourcePath = filePath;
            _fileBytes = (new FileInfo(filePath)).Length;
        }

        public void Sendfile(Socket sock)
        { 
            //convert the filesize into a byte array, reverse if it is little endian
            byte[] byteFileSize = BitConverter.GetBytes(_fileBytes); 
            if (BitConverter.IsLittleEndian) 
                Array.Reverse(byteFileSize); 
       
            //Let the client know the size of the file with a 4 byte 
            //array representing an int which is the file byte count
            sock.Send(byteFileSize); 

            //send the individual file pieces, loading only 1 into memory at a time
            for (int i = 0; i < _fileBytes; i+=_PieceSize)
                sock.Send(FileTools.GetData(_sourcePath, i, _PieceSize));
        }
    }

    private static class FileTools
    {
        //Get a chunk of data from a file
        public static byte[] GetData(string sPath, int iOffset, int iSize)
        {
            //make a filestream to the source file
            FileStream fs = new FileStream(sPath, FileMode.Open);
            //build a byte array for the data
            byte[] _data = new byte[iSize];
            //read the data and determine the amount of data actually loaded
            int iTempLength = fs.Read(_data, iOffset, iSize);
            //if the amount loaded is equal to the size arg, return the loaded data
            if (iTempLength == iSize)
                return _data;
            //if the piece is smaller than the given size arg resize the data array
            byte[] dataResized = new byte[iTempLength];
            Array.Copy(_data, dataResized, iTempLength);
            return dataResized;
        }
    }
}

[WildBamaBoy]

Solved. Thanks for your help everyone.