sending files over TCP

[Diamonddrake]

I recently started messing with tcp socket programming. I hacked up a little instant messaging application that seems to work has some little problems but the world isn't in need of more IM clients. but I wanted to know a good practice for sending large files over tcp sockets.

i made a simple client server app pair that will let the client send messages to the server. I have the server check the messages for special strings to tell it how to respond, if "#FILE" is sent it knows to write the next stream byte array to disk as a file. and this works great for a 25kb image. but as for a large file. how is this done?

also, how would progress data be sent?

i currently only know how to send the fully byte array in a single chuck, and I can't seem to find any GOOD, documented code examples.

if anyone knows and good links or code examples I would appreciate it. I would like to create a a simple app that runs on the system tray that would let me and my friends easily transfer files of any size back and fourth. there are a BILLION google result titles "C# sending files with TCP." but I looked at hundreds, the were either poorly written with no understandable comments, or they were other people asking my question with no answers.

[ddanbe]

I have this book at home maybe it helps.

[Diamonddrake]

Thanks danny! That's a great link. nothing like a free book on the subject!. I read through a bit. i will delve into it as soon as I get a chance.

[ddanbe]

Here and there perhaps a bit outdated I intend to something with it myself. If I find the time ;)

[sknake]

There are two ways of going about this too. FTP as an example implements both.

FTP Active -- You connect to the FTP on :21 and have a "command" session. When you request a file download or send it opens up another port and sends that information to the client. The client then connects to that port and gets/sends the file.

FTP Passive -- You connect to the FTP on :21 and when you request a file the "command" session sends or receives the file. This doesn't require two connections but also does not allow you to move around the FTP site while files are processing since the command connection is tied up with file xfer'ing.

The first method is ideal but sometimes due to router & networking issues the ports may not be set up properly so you can switch to passive to get the data. I'm not sure if you even have a command connection to begin with but I just wanted to point that out in case it makes a difference

[Diamonddrake]

not sure about using ftp. I don't know about creating a ftp server, I was curious about how to send messages and files simultaneously I assumed I would have to write a second thread that listened on another port. but I am using TCP and communicating via network streams. Both my clients and servers are desktop applications.

[sknake]

Well thats what I was getting at. If you want to send messages that is identical to the "control port" over FTP, and the files over the secondary ports. The problem is if you use one port then while a file is sending you cannot send messages -- or else the message text would cause the file to get corrupted. If you use 2 ports then you can send & receive files and messages on different sockets and life is good EXCEPT when firewalls don't have port forwarding set up properly which is a very common problem.

You have to choose which way you want it to work, or try to support both. This is the same issue as FTP active/passive ;) Today you will notice that with most instant messengers everyone connects to a central server which routes all of the messages. Way back when ICQ used to do p2p where you would connect to the other persons computer to send them an IM. As people started implementing firewalls this became very problematic so it is almost standard to use a central operating server for no purpose other than to switch packets.

[Diamonddrake]

first, Danny I read that entire book today, its full of good stuff. but it didn't exactly answer my question, because when it got to the complicated stuff such as sending files it moved over into concept, just telling you that you need to close the stream to tell the server the file send is over, and have the file sent in a separate thread and not actually going into any details. where are most of the book explains conventions of sending messages and getting connection information. regardless I still know much more about socket programming then I did previously, for this I thank you.

and as to you scott. I have my home network set up for 2 of my systems to have static IPs and my router forwards the port I am using to one of my static machines that I am using for my server. (I also use that machine for torrents, just on a different port) the code I am currently using all the clients connect to that machine. (I have a much worse issue now that my DSL IP address changes about 3 times a day so I am constantly updating it in my client code while testing)

but that is trivial to the current problem at hand.
I can't seem to send data in parts and reassembly it.
I can only achieve sending a small piece of data in entirety.

ex:

//netStream is a network stream object from a tcpClient connection
//this sends the entire byte array all at once.
                byte[] outStream = System.Text.Encoding.ASCII.GetBytes("some text to send");
                netStream.Write(outStream, 0, outStream.Length);
                netStream.Flush();

//server side
//clientSocket here is a TCPClient object
//fills bytearray with stream data in one read

 byte[] bytesFrom = new byte[10240];//10k byte array

networkStream.Read(bytesFrom, 0, (int)clientSocket.ReceiveBufferSize);

the data here is text but I would like to send a file
I would like to know how I could send the data in like 10kb(or larger) chunks and recieve it at the server and put it back together.

[sknake]

Regardless of the size of the packet you want to send it can only be as large as the MTU of your NIC
As seen:

sk@sk:~$ /sbin/ifconfig eth0 | grep -i mtu
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

Running 1500 is standard. The next most common is <1500, the least common is >1500 (called jumbo frames). The purpose of explaining all of that was to hopefully show that the chunk size doesn't really matter -- let the OS handle it.

Here is the client code:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;
using System.Net.Sockets;
using System.IO;
using System.Runtime.Remoting.Messaging;

namespace Client
{
  public delegate void OnSendFileCompleteDelegate(object sender, SendFileCompleteEventArgs e);
  public class SendFileCompleteEventArgs : EventArgs
  {
    private string _fileName;
    private DateTime _started;
    private DateTime _completed;
    public string FileName { get { return _fileName; } }
    public DateTime Started
    {
      get { return _started; }
      internal set { _started = value; }
    }
    public DateTime Completed
    {
      get { return _completed; }
      internal set { _completed = value; }
    }
    public TimeSpan TimeTaken
    {
      get
      {
        return this.Completed.Subtract(this.Started);
      }
    }

    private SendFileCompleteEventArgs()
      : this(string.Empty)
    {
    }
    public SendFileCompleteEventArgs(string FileName)
    {
      this._fileName = FileName;
    }
  }

  public class ClientStuff
  {
    public const int DEFAULT_SERVER_PORT = 30043;
    private IPEndPoint endPoint;

    public event OnSendFileCompleteDelegate OnSendFileComplete;

    private ClientStuff()
    {
    }
    public ClientStuff(string IP, int Port)
      : this(IPAddress.Parse(IP), Port)
    {
    }
    public ClientStuff(IPAddress IP, int Port)
      : this(new IPEndPoint(IP, Port))
    {
    }
    public ClientStuff(IPEndPoint EndPoint)
      : this()
    {
      if (EndPoint == null)
        throw new ArgumentNullException("EndPoint");

      this.endPoint = EndPoint;
    }
    public void SendFile(string FileName)
    {
      new Func<string, SendFileCompleteEventArgs>(SendFileWorker).BeginInvoke(
        FileName,
        new AsyncCallback(SendFileCallback),
        null);
    }

    private SendFileCompleteEventArgs SendFileWorker(string FileName)
    {
      SendFileCompleteEventArgs result = null;

      using (FileStream fs = new FileStream(FileName, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
      {
        using (TcpClient cli = new TcpClient())
        {
          cli.Connect(this.endPoint);
          result = new SendFileCompleteEventArgs(FileName);
          result.Started = DateTime.Now;

          using (NetworkStream ns = cli.GetStream())
          {
            StreamHelper.CopyStreamToStream(fs, ns, null);
            ns.Flush();
            ns.Close();
          }
          result.Completed = DateTime.Now;
          return result;
        }
      }
    }
    private void SendFileCallback(IAsyncResult ar)
    {
      AsyncResult result = (AsyncResult)ar;
      Func<string, SendFileCompleteEventArgs> del = (Func<string, SendFileCompleteEventArgs>)result.AsyncDelegate;
      try
      {
        SendFileCompleteEventArgs args = del.EndInvoke(ar);
        if (args != null)
        {
          var evt = this.OnSendFileComplete;
          if (evt != null)
            evt(this, args);
        }
      }
      catch
      {
        throw; //handle this
      }
    }
  }
}

Client caller:

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using System.Net;

namespace Client
{
  public partial class frmClient : Form
  {
    private ClientStuff client;

    public frmClient()
    {
      InitializeComponent();
    }

    private void frmClient_Load(object sender, EventArgs e)
    {
      StartClient();
      this.Location = new Point(0, 0);
    }

    private void StartClient()
    {
      if (client != null)
        throw new InvalidOperationException("Client is already running");

      client = new ClientStuff(IPAddress.Loopback, ClientStuff.DEFAULT_SERVER_PORT);
      client.OnSendFileComplete += new OnSendFileCompleteDelegate(client_OnSendFileComplete);
    }

    private void StopClient()
    {
      //I didn't add stop support for the client
      if (client != null)
      {
        client = null;
      }
    }

    void client_OnSendFileComplete(object sender, SendFileCompleteEventArgs e)
    {
      if (this.InvokeRequired)
      {
        this.Invoke(new OnSendFileCompleteDelegate(client_OnSendFileComplete), sender, e);
      }
      else
      {
        MessageBox.Show(
          string.Format("Sent file {0} in {1}", e.FileName, e.TimeTaken), 
          this.Text, 
          MessageBoxButtons.OK, 
          MessageBoxIcon.Information);
      }
    }

    private void button1_Click(object sender, EventArgs e)
    {
      if (openFileDialog1.ShowDialog() == DialogResult.OK)
      {
        client.SendFile(openFileDialog1.FileName);
      }
    }

  }
}

Server Code:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net.Sockets;
using System.Net;
using System.IO;
using System.Threading;
using System.Runtime.Remoting.Messaging;

namespace Server
{
  #region event stuff
  public delegate void OnFileReceivedDelegate(object sender, OnFileReceivedArgs e);
  public class OnFileReceivedArgs : EventArgs
  {
    private byte[] _buffer;
    public byte[] Buffer { get { return _buffer; } }

    private OnFileReceivedArgs()
      : base()
    {
    }
    public OnFileReceivedArgs(byte[] buffer)
      : this()
    {
      this._buffer = buffer;
    }
  }
  #endregion

  public class ServerStuff : IDisposable
  {
    public const int DEFAULT_SERVER_PORT = 30043;

    private TcpListener listener;
    private List<WeakReference> connections; //dont want our handy conn list to stop garbage collection

    public event OnFileReceivedDelegate OnFileReceived; 

    public int ConnectionCount
    {
      get
      {
        lock (connections)
        {
          return connections.Count;
        }
      }
    }

    private ServerStuff()
      : this(DEFAULT_SERVER_PORT)
    {
    }
    public ServerStuff(int Port)
    {
      connections = new List<WeakReference>();
      listener = new TcpListener(IPAddress.Any, DEFAULT_SERVER_PORT);
    }

    public void Start()
    {
      listener.Start();
      listener.BeginAcceptSocket(new AsyncCallback(ConnectionCallback), listener);
    }
    public void Stop()
    {
      listener.Stop();
      KillRunningThreads();
    }

    private void TrackThread()
    {
      lock (connections)
      {
        connections.Add(new WeakReference(Thread.CurrentThread, false));
      }
    }
    private void RemoveThread()
    {
      Thread t = Thread.CurrentThread;
      lock (connections)
      {
        for (int i1 = 0; i1 < connections.Count; i1++)
        {
          if ((connections[i1].Target as Thread) == t)
          {
            connections.RemoveAt(i1);
            break;
          }
        }
      }
    }
    private Thread[] GetRunningThreads()
    {
      lock (connections)
      {
        List<Thread> result = new List<Thread>();

        foreach (WeakReference wr in connections)
        {
          if (wr.IsAlive)
            result.Add((Thread)wr.Target);
        }

        return result.ToArray();
      }
    }

    private void KillRunningThreads()
    {
      Thread[] threads = GetRunningThreads();
      foreach (Thread t in threads)
      {
        try
        {
          t.Abort();
        }
        catch { }
      }
    }

    private void ConnectionCallback(IAsyncResult ar)
    {
      TcpListener listener = (TcpListener)ar.AsyncState;
      try
      {
        Socket s = listener.EndAcceptSocket(ar);
        new Func<Socket, byte[]>(HandleSocketComms).BeginInvoke(
          s,
          new AsyncCallback(HandleSocketCommsCallback),
          s);
      }
      catch
      {
        //You should handle this but this should be a _rare_ error
        throw;
      }
      finally
      {
        //Prime up the listener to accept another inbound request
        listener.BeginAcceptSocket(new AsyncCallback(ConnectionCallback), listener);
      }
    }

    /// <summary>
    /// This basically blocks until the socket closes and we have
    /// received all of the data
    /// </summary>
    /// <param name="s"></param>
    /// <returns></returns>
    private byte[] HandleSocketComms(Socket s)
    {
      //Dont do any exception handling here
      TrackThread();

      using (MemoryStream ms = new MemoryStream()) //This may be a bad idea for big files :)
      {
        do
        {
          int szToRead = s.Available;
          byte[] buffer = new byte[szToRead];
          
          int szRead = s.Receive(buffer, szToRead, SocketFlags.None);
          if (szRead > 0)
          {
            ms.Write(buffer, 0, szRead);
            Console.WriteLine("Read " + szRead.ToString());
          }
        }
        while (SocketConnected(s));

        return ms.ToArray();
      }
      //return result;
    }

    private static bool SocketConnected(Socket s)
    {
      return !(s.Poll(1000, SelectMode.SelectRead) && (s.Available == 0));
    }

    private void HandleSocketCommsCallback(IAsyncResult ar)
    {
      byte[] file = null;
      bool error = false;

      try
      {
        AsyncResult result = (AsyncResult)ar;
        var del = (Func<Socket, byte[]>)result.AsyncDelegate;
        file = del.EndInvoke(ar);

        Socket s = (ar.AsyncState as Socket);
        if (s != null)
        {
          //Sockets throw exceptions++ when disposed
          try
          {
            s.Close(0);
            ((IDisposable)s).Dispose();
          }
          catch { }
        }
      }
      catch (ThreadAbortException) //When we stop our server
      {
        //We were killed! Just let it die out
        error = true;
      }
      catch
      {
        //Might want to handle this
        error = true;
      }
      finally
      {
        RemoveThread(); //conn is closed at this point
        if (!error && (file != null) && (file.Length > 0))
        {
          var del = this.OnFileReceived;
          if (del != null)
            del(this, new OnFileReceivedArgs(file));
        }
      }
    }

    #region IDisposable Members
    public void Dispose()
    {
      if (listener != null)
      {
        listener.Stop();
        listener = null;
      }
      KillRunningThreads();
    }
    #endregion
  }
}

Server caller:

using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Linq;
using System.Text;
using System.Windows.Forms;
using System.IO;
using System.Diagnostics;

namespace Server
{
  public partial class frmServer : Form
  {
    private ServerStuff server;

    public frmServer()
    {
      InitializeComponent();
    }

    private void StartServer()
    {
      if (server != null)
        throw new InvalidOperationException("Server is already running");

      server = new ServerStuff(ServerStuff.DEFAULT_SERVER_PORT);
      server.Start();
      server.OnFileReceived += new OnFileReceivedDelegate(server_OnFileReceived);
    }

    private void StopServer()
    {
      if (server != null)
      {
        server.OnFileReceived -= new OnFileReceivedDelegate(server_OnFileReceived);
        server.Stop();
       ((IDisposable)server).Dispose();
       server = null;
      }
    }

    void server_OnFileReceived(object sender, OnFileReceivedArgs e)
    {
      if (this.InvokeRequired)
      {
        //Just in case we want to play with the UI
        this.Invoke(new OnFileReceivedDelegate(server_OnFileReceived), sender, e);
      }
      else
      {
        string tmpFile = Path.GetTempFileName();
        #region change the extension
        {
          string newFile = Path.ChangeExtension(tmpFile, ".bmp");
          File.Move(tmpFile, newFile);
          tmpFile = newFile;
        }
        #endregion

        File.WriteAllBytes(tmpFile, e.Buffer);
        Process p = Process.Start(tmpFile);
        p.WaitForExit();
        File.Delete(tmpFile);
      }
    }

    

    private void frmServer_Load(object sender, EventArgs e)
    {
      StartServer();
      this.Location = new Point(SystemInformation.WorkingArea.Width - this.Width, 0);
    }

    private void button1_Click(object sender, EventArgs e)
    {
      //byte[] b = File.ReadAllBytes(@"C:\img\dw\desktop.bmp");
      //System.Diagnostics.Debugger.Break();
      //6912054
    }
  }
}

I didn't implement full stop functionality but I partially plumbed it and the KillThreads() code is untested but it should work.

[sknake]

A little more on frame size:

The standard MTU for computers today is 1500 as mentioned earlier. The TCP Header size is 32 bytes so you can really send (1500-32)=1468 bytes of data in a packet. The header contains the packet's routing information.

By decreasing your MTU you can force the number of packets up. If you wanted to send 2900 bytes of data:
1500MTU : 2900/(1500-32)=1.97547683 (2 packets total, round up)
750 MTU: 2900/(750-32)=4.0389972 (5 packets total, round up)
What is the advantage of forcing the packet count up? Traffic Shaping!
If you want to limit bandwidth to X Mbps you have more fine-grain control by holding up a smaller packet than a larger packet. So by increasing the count and decreasing the payload it gives you more control to regulate your total throughput.

The problems: Going back to the fact that most computers operate with 1500 MTU if your MTU is 750 and you go a website, say, www.google.com and their MTU is 1500 you will have a problem:
1500byte google packet -> 750 MTU machine. Your machine can't receive the packet so it says "Wait, That is too big!" and it will send Google back an ICMP fragmentation-needed message. Google will then fragment the packet to the size specified in the ICMP message and send it back in a size that you can read.

Now that should work, right? No. Let me quote the iptables man page:

TCPMSS
This target allows to alter the MSS value of TCP SYN packets, to control the maximum size for that connection (usually limiting it to your outgoing interface's MTU minus 40). Of course, it can only be used in conjunction
with -p tcp. It is only valid in the mangle table.
This target is used to overcome criminally braindead ISPs or servers which block ICMP Fragmentation Needed packets. The symptoms of this problem are that everything works fine from your Linux firewall/router, but machines
behind it can never exchange large packets:
1) Web browsers connect, then hang with no data received.
2) Small mail works fine, but large emails hang.
3) ssh works fine, but scp hangs after initial handshaking.
Workaround: activate this option and add a rule to your firewall configuration like:
iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
-j TCPMSS --clamp-mss-to-pmtu

So if you're running 750 and google is running 1500 and someone in the middle of you two drops the ICMP Fragmentation packet then you're screwed. You won't be able to communicate with the remote server. You can set up your edge device to clamp outbound packets to the MTU discovery by Path MTU Discovery (PMTU) but this also does not work all of the time as expected.

Likewise all of this works in reverse for jumbo frames BUT if you have a larger MTU than sites on the internet it won't be a problem:
1500MTU Google >> 50000000MTU machine will make your machine laugh and say "give me more data"

I have read that some people claim you can communicate with LAN appliances better with jumbo frames BUT the biggest bottleneck in TCP/IP is usually disk I/O speed on either end. 99% of the time someone screws with an MTU it breaks something.

You should be ready to start writing network applications now :)

[edit]
My math was off. The total TCP header size can be up to 40 bytes, not just 32. That is insignificant in the points made though.
[/edit]

[Diamonddrake]

wow, thanks sknake, i decided on 1024 bytes for my file packet read and writes. I seem to get ok results with it. I am going to spend a few reviewing your code and see what I can make sense of it. It appears you are using remoting, but its hard to tell, since I am new to network programming.

I just figured out a simple way to loop through the bytes of a file on disk and send it out with loop, and to read it with a similar loop. the trick was to send the length of the file in bytes to the server first. what I ended up doing was sending "#FILE:numerofbytes" to the server, when a message was received from the client it checks if it starts with a "#", if so it passes the message into my command processing method, where it checks if it starts with "#FILE" if so it splits it at the ":" and parses the numbers at the end into a long int and then I use that information to start a loop that first sends a message saying its ok to send the file, ("#RFFS") then starts the read, the client then gets the message and starts the loop of sending the file in 1024 byte chunks. the client loops and sends, the server loops and revives, part of the loop writes the bytes as they are received to a file stream its really choppy but it works, and allows for an easy durring loop calculation of percentage sent/recived on both sides.

Question1: - Is my solution an acceptable one?
or should I really use the asynchronous begin and end send file methods of the socket class? (I had trouble figuring them out)

[sknake]

No that code doesn't use remoting, just some of the messaging functionality for callbacks.

You should probably learn Asynch sockets since you don't want to block your main thread while it is processing and Async vs sync is a different code model in the back end so if you build on top of it you may have to refactor more code.

Take a look at the code I posted and see if it makes sense.

[sknake]

Oh also -- the code I posted is all asynchronous so you can send files concurrently

client.SendFile("1");
client.SendFile("2");
client.SendFile("3");

And it will have three file transmissions occuring concurrently on three separate sockets. It will fire off events as each file xfer completes.

[Diamonddrake]

I loaded up your example solution, ran the apps and it works, I can say that :). looked through the code, and it looks beautiful. Much better than my chicken scratch. I understand most of it. I really like the custom event argument approach. Its a lot of code to take in. I am not up on threading, I understand how to create threads, and I understand calling methods on the main thread from another thread needs to be invoked... other than that. yeah... so the terms like callback throw me off, and the methods formed like this one:

private void ConnectionCallback(IAsyncResult ar)
    {
      TcpListener listener = (TcpListener)ar.AsyncState;
      try
      {
        Socket s = listener.EndAcceptSocket(ar);
        new Func<Socket, byte[]>(HandleSocketComms).BeginInvoke(
          s,
          new AsyncCallback(HandleSocketCommsCallback),
          s);
      }
      catch
      {
        //You should handle this but this should be a _rare_ error
        throw;
      }
      finally
      {
        //Prime up the listener to accept another inbound request
        listener.BeginAcceptSocket(new AsyncCallback(ConnectionCallback), listener);
      }
    }

that "new func<socket, byte[]>" line I have never seen before.

the client code seems straight forward, but if you get some free time and feel like it. I sure would like a little more explination on the server code. But only if you have time. Thanks alot.

I tested my 2 little loops on a 409mb file, and it works great on a local network. Im about to test yours with the same file and see how well it works. Thanks again, you are awesome!

[Diamonddrake]

your client said my 409mb video file was sent successfully in 50 seconds. but the server side windows first ran out of virtual memory and increased its default paging size. not sure why because that system has 2 gigs of ram) anyhow. then when your app auto open windows media player to play the video file, (I modified the file extention form bmp to the .avi ext) windows media player gave an error message box with soemthing in hex) and exited. I tried to modify your server to use a filestream instead of a memory stream but I was unsuccessful. as exactly what that memory stream was doing is unclear to me.

[Diamonddrake]

Sorry, Im back. I understand that the memory stream is being returned, I just don't see where its getting sent to, I understand from the event handler code on the form that the memory stream somehow gets sent as an event object, I just don't see how. I am sure I will eventually figure it out. maybe after a break. i have been either reading about sockets, or mashing up socket code for a good 7 hours now and i think after a good break it will make more sense. I appreciate it sknake. you are a great asset to this site!, (you too danny, thanks again for the pdf);

[Diamonddrake]

Visual studio's intelisense gave me all the information I needed to know to change the return type of the HandleSocketComms method and change the the memory stream to a file stream, it all works well except using this method, I can't seem to figure out how to send the file extension from the client to the server.

I think I can extend these classes you have provided to do great things for my applications. I can't say it enough. I appreciate the help.

[sknake]

Yes -- I did know it wouldn't work for large files that is why i put:

using (MemoryStream ms = new MemoryStream()) //This may be a bad idea for big files :)

For large files your solution is to change that memorystream to a file stream. Use Path.GetTempFile() and start piping the contents to a file instead of memory. You would also want to change the event to pass the string of the temp file name instead of the byte array, and then handle the event as a file.

The application I gave you was not intended to handle sending the file information over the socket, just to send the raw data asynchronously. But with a little work you could implement a command structure and a file structure on the same socket.

If you have a specific question about any of the comms let me know and maybe we can get your problem sorted out :)

[Diamonddrake]

first off I modified your custom arguments to take a string

public class OnFileReceivedArgs : EventArgs
  {
    private byte[] _buffer;
    public byte[] Buffer { get { return _buffer; } }

    private string _FilePath;
    public string FilePath { get { return _FilePath; } }

    private OnFileReceivedArgs()
      : base()
    {
    }
    public OnFileReceivedArgs(byte[] buffer)
      : this()
    {
      this._buffer = buffer;
    }

    public OnFileReceivedArgs(string filePath)
        : this()
    {
        this._FilePath = filePath;
    }
  }

then I modified your connection callback method to return a string instead of a byte array.

private void ConnectionCallback(IAsyncResult ar)
    {
      TcpListener listener = (TcpListener)ar.AsyncState;
      try
      {
        Socket s = listener.EndAcceptSocket(ar);
        new Func<Socket, string>(HandleSocketComms).BeginInvoke(
          s,
          new AsyncCallback(HandleSocketCommsCallback),
          s);

       
      }
      catch
      {
        //You should handle this but this should be a _rare_ error
        throw;
      }
      finally
      {
        //Prime up the listener to accept another inbound request
        listener.BeginAcceptSocket(new AsyncCallback(ConnectionCallback), listener);
      }
    }

then I modified your handlesocketcomms method to save to a file stream

private string HandleSocketComms(Socket s)
    {
      //Dont do any exception handling here
      TrackThread();
      string tempfileloc = Path.GetTempFileName();
      using (FileStream ms = new FileStream(tempfileloc, FileMode.Create, FileAccess.Write)) //This may be a bad idea for big files :)
      {
        do
        {
          int szToRead = s.Available;
          byte[] buffer = new byte[szToRead];
          
          int szRead = s.Receive(buffer, szToRead, SocketFlags.None);
          if (szRead > 0)
          {
            ms.Write(buffer, 0, szRead);
            Console.WriteLine("Read " + szRead.ToString());
          }
        }
        while (SocketConnected(s));

        return tempfileloc;
      }
      //return result;
    }

Finally I modified your handlesocketcommscallback method to get the new eventarg and create the new eventargument class

private void HandleSocketCommsCallback(IAsyncResult ar)
    {
      byte[] file = null;
      string filepath = "";
      bool error = false;

      try
      {
        AsyncResult result = (AsyncResult)ar;
        var del = (Func<Socket, string>)result.AsyncDelegate;
        filepath = del.EndInvoke(ar);

        Socket s = (ar.AsyncState as Socket);
        if (s != null)
        {
          //Sockets throw exceptions++ when disposed
          try
          {
            s.Close(0);
            ((IDisposable)s).Dispose();
          }
          catch { }
        }
      }
      catch (ThreadAbortException) //When we stop our server
      {
        //We were killed! Just let it die out
        error = true;
      }
      catch
      {
        //Might want to handle this
        error = true;
      }
      finally
      {
        RemoveThread(); //conn is closed at this point
        //if (!error && (file != null) && (file.Length > 0))
        if (!error)
        {
          var del = this.OnFileReceived;
          if (del != null)
            del(this, new OnFileReceivedArgs(filepath));
        }
      }
    }

now I want to be straight forward with you. I understand the flow of code here. I see the order of code execution, and I understand the general idea. But as for the specifics. I really have no understanding of how the IAsyncresult object works, or the asynccallback stuff.

An issue for me, is I can't figure out in this scenario how I would send information like file name, extention, and length, that way I could calculate progress. In the one I did I sent text across the stream and waited for an answer on the client before I started sending the information. but with your code, I am just kinda confused with where to do this.

I do appreciate all the help. ( the file stream version did successfully transmit the 409mb file across my home network)

[sknake]

OK this is going back to my original comments about how FTP does.

Solution 1)
(client->server) I'm sending blah.txt 4096 bytes
(server->client) ok hit me up on port:30044 for the xfer
(client->server) connects and sends file

Solution 2)
(client->server) I'm sending blah.txt 4096 bytes
(client->server) sends file over existing connection

Solution 1 requires two ports (= more firewall configurations)
Solution 2 requires more overhead for inspecting the data

From our conversation I feel that you want to use solution 2, so:
I would create another assembly ClientServer.Common as a class library and define a class like:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.Xml.Serialization;

namespace ClientServer.Common
{
  public enum CommandType
  {
    SendFile
  }
  public class Command
  {
    
    /// <summary>
    /// When this byte collection is hit you know you have received a command
    /// </summary>
    public static readonly byte[] COMMAND_DELIM = new byte[] { 7, 9, 13, 10, 10, 13 }; //This is invalid inside an XML document so its "good-enough"


    public CommandType CommandCode { get; set; }
    /// <summary>
    /// File size in bytes
    /// </summary>
    public int FileSize { get; set; }
    public string FileName { get; set; }
    public Command()
    {
    }

    public byte[] Serialize()
    {
      using (MemoryStream ms = new MemoryStream())
      {
        XmlSerializer ser = new XmlSerializer(typeof(Command));
        ser.Serialize(ms, this);
        return ms.ToArray();
      }
    }

    public static Command FromBuffer(byte[] buffer)
    {
      using (MemoryStream ms = new MemoryStream(buffer))
      {
        return FromStream(ms);
      }
    }
    public static Command FromStream(Stream s)
    {
      s.Seek(0, SeekOrigin.Begin);
      s.Position = 0;
      XmlSerializer ser = new XmlSerializer(typeof(Command));
      return (Command)ser.Deserialize(s);
    }
  }
}

Now you can't have undisplayable characters and line feeds inside the body of an XML document or it will blow up the DOM parser in the .NET framework so our signature is safe in that regard for delimiting between a command and file. You would create the signature and send a packet like:

client->server
(command.Serialize()) + COMMAND_DELIM + FILE_DATA

What the server would do is this (you will have to undo the work you previously did):
In the HandleSocksComms it would need to buffer the inbound data in a memory stream again but this time we won't let it run the system out of memory.

As data is received it needs to add the buffer to the memory stream
After the data is appending it needs to check to see if the signature is found
> If the signature is found then send all byte[]s before the signature to static methods of the command class to parse the serialized data
> If the signature is not found then continue buffering the input
> If the buffered input is greater than 10KB then kill the socket because someone has connected to you and is just sending you crap. The serialized command class as it stands will be very small

So to visually represent the packet you could see it like this
C=Command, D=Delimiter, F=File
CCCCCCDDDDDDFFFFFFFFFFFFFFF

You can't check the indivual buffer you just received for the signature because it is possible that the packets will fragment across signature, like so:
PACKET1 | PACKET2
CCCCCCDDD | DDDFFFFFFFFFFFFFFF

Now if you fail to deserialize the class then kill the socket because its some unknown software on the other end, again, sending crap.

What you could do is look at the XML DOM's specs and see if there is any single character that is not allow in XML (I know there is) and use that single byte as your delimiter. IE if you can't have a null (=0 value) then you could separate them like:
CCCCC0FFFFF

And since a single byte cannot be fragmented across the sends you wouldn't have to check the in-memory buffer and life will be easier separating between commands and files.

Now once you have the command deserialized in memory you will read the stream the exact length of the file as indicated in the Command's FileSize property. If the buffer is smaller then throw it all away, the transmission did not complete. After you have read the exact length of the file you will be in "waiting for command" mode again where you will continue to read until you hit a sig and attempt to deserialize the signature.

You cannot rely on a command delimiter after the binary file because say your deserialized command class says the file size is 4000 bytes, but the client actually sends 4500 bytes of data -- it could have whatever delimiter you specify in the file because it is binary data and any byte value is potentially valid. Example
CCCCCDDDFFFFFFFFFDDDFFFF
In this case the file, as a matter of luck, just so happens to contain your signature and your application would break when someone sends this file. This would lead to a very hard to diagnose issue.

[disclaimer]
Now -- All of these concepts above are perfectly valid and will work but they may not be the most efficient. They are very good for academic purposes or even an application with low-moderate usage. If thousands of users are going to use this app then every bit matters and this would be seen as a horribly ineffecient waste of space, bandwidth, memory, and disk space. If we really wanted to be high performance we would use less than byte to transmit the command and mask the bits out on either end to keep storage space to a minimal but this gets a little complicated. Plus we wouldn't use the XmlSerializer because that is a little slow...
[/disclaimer]

[Diamonddrake]

Ok, I kinda get what you are talking about, but Im not sure how to do that. your client

private SendFileCompleteEventArgs SendFileWorker(string FileName)
    {
      SendFileCompleteEventArgs result = null;

      using (FileStream fs = new FileStream(FileName, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
      {
        using (TcpClient cli = new TcpClient())
        {
          cli.Connect(this.endPoint);
          result = new SendFileCompleteEventArgs(FileName);
          result.Started = DateTime.Now;

          using (NetworkStream ns = cli.GetStream())
          {
            StreamHelper.CopyStreamToStream(fs, ns, null);
            ns.Flush();
            ns.Close();
          }
          result.Completed = DateTime.Now;
          return result;
        }
      }
    }

Problem: I just can't see how to add your serialized data to that.

Problem: or even how to parse that on the otherside.

Im trying, this just really isn't my area. Also, I wouldn't mind using additional ports. as my desktop is the server for this, and working with firewalls and routers is simple.

This isn't going to be a mass used application, I intend to use it for 2 purposes. 1) to share files with a couple of my friends, and 2) to have a reliable way to send/get data to and from my desktop when I am not home.

again thanks for the help.

[sknake]

No problem. I have exams coming up so it might be a few days before I can get back to you on this.

[Diamonddrake]

That's alright :) I am studding for the compTIA A+ exams myself. I will continue to attempt this and if I get any where with it I will post my results. Otherwise, when you get a chance, I would appreciate some more assistance. but if you don't get back to it I understand. you have been a big help.

Thanks again.

[sknake]

Next monday fire up a new thread and upload the project you have so far and we'll go from there.

Good luck on your exam!