| | |
sending files over TCP
Please support our C# advertiser: Intel Parallel Studio Home
Thread Solved |
I recently started messing with tcp socket programming. I hacked up a little instant messaging application that seems to work has some little problems but the world isn't in need of more IM clients. but I wanted to know a good practice for sending large files over tcp sockets.
i made a simple client server app pair that will let the client send messages to the server. I have the server check the messages for special strings to tell it how to respond, if "#FILE" is sent it knows to write the next stream byte array to disk as a file. and this works great for a 25kb image. but as for a large file. how is this done?
also, how would progress data be sent?
i currently only know how to send the fully byte array in a single chuck, and I can't seem to find any GOOD, documented code examples.
if anyone knows and good links or code examples I would appreciate it. I would like to create a a simple app that runs on the system tray that would let me and my friends easily transfer files of any size back and fourth. there are a BILLION google result titles "C# sending files with TCP." but I looked at hundreds, the were either poorly written with no understandable comments, or they were other people asking my question with no answers.
i made a simple client server app pair that will let the client send messages to the server. I have the server check the messages for special strings to tell it how to respond, if "#FILE" is sent it knows to write the next stream byte array to disk as a file. and this works great for a 25kb image. but as for a large file. how is this done?
also, how would progress data be sent?
i currently only know how to send the fully byte array in a single chuck, and I can't seem to find any GOOD, documented code examples.
if anyone knows and good links or code examples I would appreciate it. I would like to create a a simple app that runs on the system tray that would let me and my friends easily transfer files of any size back and fourth. there are a BILLION google result titles "C# sending files with TCP." but I looked at hundreds, the were either poorly written with no understandable comments, or they were other people asking my question with no answers.
2
#2 Oct 9th, 2009
I have this book at home maybe it helps.
Today is a gift, that's why it is called "The Present".
Make love, no war. Cave ab homine unius libri.
Danny
Make love, no war. Cave ab homine unius libri.
Danny
0
#5 Oct 10th, 2009
There are two ways of going about this too. FTP as an example implements both.
FTP Active -- You connect to the FTP on :21 and have a "command" session. When you request a file download or send it opens up another port and sends that information to the client. The client then connects to that port and gets/sends the file.
FTP Passive -- You connect to the FTP on :21 and when you request a file the "command" session sends or receives the file. This doesn't require two connections but also does not allow you to move around the FTP site while files are processing since the command connection is tied up with file xfer'ing.
The first method is ideal but sometimes due to router & networking issues the ports may not be set up properly so you can switch to passive to get the data. I'm not sure if you even have a command connection to begin with but I just wanted to point that out in case it makes a difference
FTP Active -- You connect to the FTP on :21 and have a "command" session. When you request a file download or send it opens up another port and sends that information to the client. The client then connects to that port and gets/sends the file.
FTP Passive -- You connect to the FTP on :21 and when you request a file the "command" session sends or receives the file. This doesn't require two connections but also does not allow you to move around the FTP site while files are processing since the command connection is tied up with file xfer'ing.
The first method is ideal but sometimes due to router & networking issues the ports may not be set up properly so you can switch to passive to get the data. I'm not sure if you even have a command connection to begin with but I just wanted to point that out in case it makes a difference
0
#6 Oct 10th, 2009
not sure about using ftp. I don't know about creating a ftp server, I was curious about how to send messages and files simultaneously I assumed I would have to write a second thread that listened on another port. but I am using TCP and communicating via network streams. Both my clients and servers are desktop applications.
0
#7 Oct 10th, 2009
Well thats what I was getting at. If you want to send messages that is identical to the "control port" over FTP, and the files over the secondary ports. The problem is if you use one port then while a file is sending you cannot send messages -- or else the message text would cause the file to get corrupted. If you use 2 ports then you can send & receive files and messages on different sockets and life is good EXCEPT when firewalls don't have port forwarding set up properly which is a very common problem.
You have to choose which way you want it to work, or try to support both. This is the same issue as FTP active/passive
Today you will notice that with most instant messengers everyone connects to a central server which routes all of the messages. Way back when ICQ used to do p2p where you would connect to the other persons computer to send them an IM. As people started implementing firewalls this became very problematic so it is almost standard to use a central operating server for no purpose other than to switch packets.
You have to choose which way you want it to work, or try to support both. This is the same issue as FTP active/passive
Today you will notice that with most instant messengers everyone connects to a central server which routes all of the messages. Way back when ICQ used to do p2p where you would connect to the other persons computer to send them an IM. As people started implementing firewalls this became very problematic so it is almost standard to use a central operating server for no purpose other than to switch packets. Last edited by sknake; Oct 10th, 2009 at 4:29 pm.
0
#8 Oct 10th, 2009
first, Danny I read that entire book today, its full of good stuff. but it didn't exactly answer my question, because when it got to the complicated stuff such as sending files it moved over into concept, just telling you that you need to close the stream to tell the server the file send is over, and have the file sent in a separate thread and not actually going into any details. where are most of the book explains conventions of sending messages and getting connection information. regardless I still know much more about socket programming then I did previously, for this I thank you.
and as to you scott. I have my home network set up for 2 of my systems to have static IPs and my router forwards the port I am using to one of my static machines that I am using for my server. (I also use that machine for torrents, just on a different port) the code I am currently using all the clients connect to that machine. (I have a much worse issue now that my DSL IP address changes about 3 times a day so I am constantly updating it in my client code while testing)
but that is trivial to the current problem at hand.
I can't seem to send data in parts and reassembly it.
I can only achieve sending a small piece of data in entirety.
ex:
the data here is text but I would like to send a file
I would like to know how I could send the data in like 10kb(or larger) chunks and recieve it at the server and put it back together.
and as to you scott. I have my home network set up for 2 of my systems to have static IPs and my router forwards the port I am using to one of my static machines that I am using for my server. (I also use that machine for torrents, just on a different port) the code I am currently using all the clients connect to that machine. (I have a much worse issue now that my DSL IP address changes about 3 times a day so I am constantly updating it in my client code while testing)
but that is trivial to the current problem at hand.
I can't seem to send data in parts and reassembly it.
I can only achieve sending a small piece of data in entirety.
ex:
C# Syntax (Toggle Plain Text)
//netStream is a network stream object from a tcpClient connection //this sends the entire byte array all at once. byte[] outStream = System.Text.Encoding.ASCII.GetBytes("some text to send"); netStream.Write(outStream, 0, outStream.Length); netStream.Flush(); //server side //clientSocket here is a TCPClient object //fills bytearray with stream data in one read byte[] bytesFrom = new byte[10240];//10k byte array networkStream.Read(bytesFrom, 0, (int)clientSocket.ReceiveBufferSize);
I would like to know how I could send the data in like 10kb(or larger) chunks and recieve it at the server and put it back together.
Last edited by Diamonddrake; Oct 10th, 2009 at 7:25 pm.
1
#9 Oct 10th, 2009
Regardless of the size of the packet you want to send it can only be as large as the MTU of your NIC
As seen:
Running 1500 is standard. The next most common is <1500, the least common is >1500 (called jumbo frames). The purpose of explaining all of that was to hopefully show that the chunk size doesn't really matter -- let the OS handle it.
Here is the client code:
Client caller:
Server Code:
Server caller:
I didn't implement full stop functionality but I partially plumbed it and the KillThreads() code is untested but it should work.
As seen:
sk@sk:~$ /sbin/ifconfig eth0 | grep -i mtu
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1Here is the client code:
C# Syntax (Toggle Plain Text)
using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Net; using System.Net.Sockets; using System.IO; using System.Runtime.Remoting.Messaging; namespace Client { public delegate void OnSendFileCompleteDelegate(object sender, SendFileCompleteEventArgs e); public class SendFileCompleteEventArgs : EventArgs { private string _fileName; private DateTime _started; private DateTime _completed; public string FileName { get { return _fileName; } } public DateTime Started { get { return _started; } internal set { _started = value; } } public DateTime Completed { get { return _completed; } internal set { _completed = value; } } public TimeSpan TimeTaken { get { return this.Completed.Subtract(this.Started); } } private SendFileCompleteEventArgs() : this(string.Empty) { } public SendFileCompleteEventArgs(string FileName) { this._fileName = FileName; } } public class ClientStuff { public const int DEFAULT_SERVER_PORT = 30043; private IPEndPoint endPoint; public event OnSendFileCompleteDelegate OnSendFileComplete; private ClientStuff() { } public ClientStuff(string IP, int Port) : this(IPAddress.Parse(IP), Port) { } public ClientStuff(IPAddress IP, int Port) : this(new IPEndPoint(IP, Port)) { } public ClientStuff(IPEndPoint EndPoint) : this() { if (EndPoint == null) throw new ArgumentNullException("EndPoint"); this.endPoint = EndPoint; } public void SendFile(string FileName) { new Func<string, SendFileCompleteEventArgs>(SendFileWorker).BeginInvoke( FileName, new AsyncCallback(SendFileCallback), null); } private SendFileCompleteEventArgs SendFileWorker(string FileName) { SendFileCompleteEventArgs result = null; using (FileStream fs = new FileStream(FileName, FileMode.Open, FileAccess.Read, FileShare.ReadWrite)) { using (TcpClient cli = new TcpClient()) { cli.Connect(this.endPoint); result = new SendFileCompleteEventArgs(FileName); result.Started = DateTime.Now; using (NetworkStream ns = cli.GetStream()) { StreamHelper.CopyStreamToStream(fs, ns, null); ns.Flush(); ns.Close(); } result.Completed = DateTime.Now; return result; } } } private void SendFileCallback(IAsyncResult ar) { AsyncResult result = (AsyncResult)ar; Func<string, SendFileCompleteEventArgs> del = (Func<string, SendFileCompleteEventArgs>)result.AsyncDelegate; try { SendFileCompleteEventArgs args = del.EndInvoke(ar); if (args != null) { var evt = this.OnSendFileComplete; if (evt != null) evt(this, args); } } catch { throw; //handle this } } } }
Client caller:
C# Syntax (Toggle Plain Text)
using System; using System.Collections.Generic; using System.ComponentModel; using System.Data; using System.Drawing; using System.Linq; using System.Text; using System.Windows.Forms; using System.Net; namespace Client { public partial class frmClient : Form { private ClientStuff client; public frmClient() { InitializeComponent(); } private void frmClient_Load(object sender, EventArgs e) { StartClient(); this.Location = new Point(0, 0); } private void StartClient() { if (client != null) throw new InvalidOperationException("Client is already running"); client = new ClientStuff(IPAddress.Loopback, ClientStuff.DEFAULT_SERVER_PORT); client.OnSendFileComplete += new OnSendFileCompleteDelegate(client_OnSendFileComplete); } private void StopClient() { //I didn't add stop support for the client if (client != null) { client = null; } } void client_OnSendFileComplete(object sender, SendFileCompleteEventArgs e) { if (this.InvokeRequired) { this.Invoke(new OnSendFileCompleteDelegate(client_OnSendFileComplete), sender, e); } else { MessageBox.Show( string.Format("Sent file {0} in {1}", e.FileName, e.TimeTaken), this.Text, MessageBoxButtons.OK, MessageBoxIcon.Information); } } private void button1_Click(object sender, EventArgs e) { if (openFileDialog1.ShowDialog() == DialogResult.OK) { client.SendFile(openFileDialog1.FileName); } } } }
Server Code:
C# Syntax (Toggle Plain Text)
using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.Net.Sockets; using System.Net; using System.IO; using System.Threading; using System.Runtime.Remoting.Messaging; namespace Server { #region event stuff public delegate void OnFileReceivedDelegate(object sender, OnFileReceivedArgs e); public class OnFileReceivedArgs : EventArgs { private byte[] _buffer; public byte[] Buffer { get { return _buffer; } } private OnFileReceivedArgs() : base() { } public OnFileReceivedArgs(byte[] buffer) : this() { this._buffer = buffer; } } #endregion public class ServerStuff : IDisposable { public const int DEFAULT_SERVER_PORT = 30043; private TcpListener listener; private List<WeakReference> connections; //dont want our handy conn list to stop garbage collection public event OnFileReceivedDelegate OnFileReceived; public int ConnectionCount { get { lock (connections) { return connections.Count; } } } private ServerStuff() : this(DEFAULT_SERVER_PORT) { } public ServerStuff(int Port) { connections = new List<WeakReference>(); listener = new TcpListener(IPAddress.Any, DEFAULT_SERVER_PORT); } public void Start() { listener.Start(); listener.BeginAcceptSocket(new AsyncCallback(ConnectionCallback), listener); } public void Stop() { listener.Stop(); KillRunningThreads(); } private void TrackThread() { lock (connections) { connections.Add(new WeakReference(Thread.CurrentThread, false)); } } private void RemoveThread() { Thread t = Thread.CurrentThread; lock (connections) { for (int i1 = 0; i1 < connections.Count; i1++) { if ((connections[i1].Target as Thread) == t) { connections.RemoveAt(i1); break; } } } } private Thread[] GetRunningThreads() { lock (connections) { List<Thread> result = new List<Thread>(); foreach (WeakReference wr in connections) { if (wr.IsAlive) result.Add((Thread)wr.Target); } return result.ToArray(); } } private void KillRunningThreads() { Thread[] threads = GetRunningThreads(); foreach (Thread t in threads) { try { t.Abort(); } catch { } } } private void ConnectionCallback(IAsyncResult ar) { TcpListener listener = (TcpListener)ar.AsyncState; try { Socket s = listener.EndAcceptSocket(ar); new Func<Socket, byte[]>(HandleSocketComms).BeginInvoke( s, new AsyncCallback(HandleSocketCommsCallback), s); } catch { //You should handle this but this should be a _rare_ error throw; } finally { //Prime up the listener to accept another inbound request listener.BeginAcceptSocket(new AsyncCallback(ConnectionCallback), listener); } } /// <summary> /// This basically blocks until the socket closes and we have /// received all of the data /// </summary> /// <param name="s"></param> /// <returns></returns> private byte[] HandleSocketComms(Socket s) { //Dont do any exception handling here TrackThread(); using (MemoryStream ms = new MemoryStream()) //This may be a bad idea for big files :) { do { int szToRead = s.Available; byte[] buffer = new byte[szToRead]; int szRead = s.Receive(buffer, szToRead, SocketFlags.None); if (szRead > 0) { ms.Write(buffer, 0, szRead); Console.WriteLine("Read " + szRead.ToString()); } } while (SocketConnected(s)); return ms.ToArray(); } //return result; } private static bool SocketConnected(Socket s) { return !(s.Poll(1000, SelectMode.SelectRead) && (s.Available == 0)); } private void HandleSocketCommsCallback(IAsyncResult ar) { byte[] file = null; bool error = false; try { AsyncResult result = (AsyncResult)ar; var del = (Func<Socket, byte[]>)result.AsyncDelegate; file = del.EndInvoke(ar); Socket s = (ar.AsyncState as Socket); if (s != null) { //Sockets throw exceptions++ when disposed try { s.Close(0); ((IDisposable)s).Dispose(); } catch { } } } catch (ThreadAbortException) //When we stop our server { //We were killed! Just let it die out error = true; } catch { //Might want to handle this error = true; } finally { RemoveThread(); //conn is closed at this point if (!error && (file != null) && (file.Length > 0)) { var del = this.OnFileReceived; if (del != null) del(this, new OnFileReceivedArgs(file)); } } } #region IDisposable Members public void Dispose() { if (listener != null) { listener.Stop(); listener = null; } KillRunningThreads(); } #endregion } }
Server caller:
C# Syntax (Toggle Plain Text)
using System; using System.Collections.Generic; using System.ComponentModel; using System.Data; using System.Drawing; using System.Linq; using System.Text; using System.Windows.Forms; using System.IO; using System.Diagnostics; namespace Server { public partial class frmServer : Form { private ServerStuff server; public frmServer() { InitializeComponent(); } private void StartServer() { if (server != null) throw new InvalidOperationException("Server is already running"); server = new ServerStuff(ServerStuff.DEFAULT_SERVER_PORT); server.Start(); server.OnFileReceived += new OnFileReceivedDelegate(server_OnFileReceived); } private void StopServer() { if (server != null) { server.OnFileReceived -= new OnFileReceivedDelegate(server_OnFileReceived); server.Stop(); ((IDisposable)server).Dispose(); server = null; } } void server_OnFileReceived(object sender, OnFileReceivedArgs e) { if (this.InvokeRequired) { //Just in case we want to play with the UI this.Invoke(new OnFileReceivedDelegate(server_OnFileReceived), sender, e); } else { string tmpFile = Path.GetTempFileName(); #region change the extension { string newFile = Path.ChangeExtension(tmpFile, ".bmp"); File.Move(tmpFile, newFile); tmpFile = newFile; } #endregion File.WriteAllBytes(tmpFile, e.Buffer); Process p = Process.Start(tmpFile); p.WaitForExit(); File.Delete(tmpFile); } } private void frmServer_Load(object sender, EventArgs e) { StartServer(); this.Location = new Point(SystemInformation.WorkingArea.Width - this.Width, 0); } private void button1_Click(object sender, EventArgs e) { //byte[] b = File.ReadAllBytes(@"C:\img\dw\desktop.bmp"); //System.Diagnostics.Debugger.Break(); //6912054 } } }
I didn't implement full stop functionality but I partially plumbed it and the KillThreads() code is untested but it should work.
0
#10 Oct 10th, 2009
A little more on frame size:
The standard MTU for computers today is 1500 as mentioned earlier. The TCP Header size is 32 bytes so you can really send (1500-32)=1468 bytes of data in a packet. The header contains the packet's routing information.
By decreasing your MTU you can force the number of packets up. If you wanted to send 2900 bytes of data:
1500MTU : 2900/(1500-32)=1.97547683 (2 packets total, round up)
750 MTU: 2900/(750-32)=4.0389972 (5 packets total, round up)
What is the advantage of forcing the packet count up? Traffic Shaping!
If you want to limit bandwidth to X Mbps you have more fine-grain control by holding up a smaller packet than a larger packet. So by increasing the count and decreasing the payload it gives you more control to regulate your total throughput.
The problems: Going back to the fact that most computers operate with 1500 MTU if your MTU is 750 and you go a website, say, www.google.com and their MTU is 1500 you will have a problem:
1500byte google packet -> 750 MTU machine. Your machine can't receive the packet so it says "Wait, That is too big!" and it will send Google back an ICMP fragmentation-needed message. Google will then fragment the packet to the size specified in the ICMP message and send it back in a size that you can read.
Now that should work, right? No. Let me quote the iptables man page:
So if you're running 750 and google is running 1500 and someone in the middle of you two drops the ICMP Fragmentation packet then you're screwed. You won't be able to communicate with the remote server. You can set up your edge device to clamp outbound packets to the MTU discovery by Path MTU Discovery (PMTU) but this also does not work all of the time as expected.
Likewise all of this works in reverse for jumbo frames BUT if you have a larger MTU than sites on the internet it won't be a problem:
1500MTU Google >> 50000000MTU machine will make your machine laugh and say "give me more data"
I have read that some people claim you can communicate with LAN appliances better with jumbo frames BUT the biggest bottleneck in TCP/IP is usually disk I/O speed on either end. 99% of the time someone screws with an MTU it breaks something.
You should be ready to start writing network applications now
[edit]
My math was off. The total TCP header size can be up to 40 bytes, not just 32. That is insignificant in the points made though.
[/edit]
The standard MTU for computers today is 1500 as mentioned earlier. The TCP Header size is 32 bytes so you can really send (1500-32)=1468 bytes of data in a packet. The header contains the packet's routing information.
By decreasing your MTU you can force the number of packets up. If you wanted to send 2900 bytes of data:
1500MTU : 2900/(1500-32)=1.97547683 (2 packets total, round up)
750 MTU: 2900/(750-32)=4.0389972 (5 packets total, round up)
What is the advantage of forcing the packet count up? Traffic Shaping!
If you want to limit bandwidth to X Mbps you have more fine-grain control by holding up a smaller packet than a larger packet. So by increasing the count and decreasing the payload it gives you more control to regulate your total throughput.
The problems: Going back to the fact that most computers operate with 1500 MTU if your MTU is 750 and you go a website, say, www.google.com and their MTU is 1500 you will have a problem:
1500byte google packet -> 750 MTU machine. Your machine can't receive the packet so it says "Wait, That is too big!" and it will send Google back an ICMP fragmentation-needed message. Google will then fragment the packet to the size specified in the ICMP message and send it back in a size that you can read.
Now that should work, right? No. Let me quote the iptables man page:
•
•
•
•
TCPMSS
This target allows to alter the MSS value of TCP SYN packets, to control the maximum size for that connection (usually limiting it to your outgoing interface's MTU minus 40). Of course, it can only be used in conjunction
with -p tcp. It is only valid in the mangle table.
This target is used to overcome criminally braindead ISPs or servers which block ICMP Fragmentation Needed packets. The symptoms of this problem are that everything works fine from your Linux firewall/router, but machines
behind it can never exchange large packets:
1) Web browsers connect, then hang with no data received.
2) Small mail works fine, but large emails hang.
3) ssh works fine, but scp hangs after initial handshaking.
Workaround: activate this option and add a rule to your firewall configuration like:
iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
-j TCPMSS --clamp-mss-to-pmtu
Likewise all of this works in reverse for jumbo frames BUT if you have a larger MTU than sites on the internet it won't be a problem:
1500MTU Google >> 50000000MTU machine will make your machine laugh and say "give me more data"
I have read that some people claim you can communicate with LAN appliances better with jumbo frames BUT the biggest bottleneck in TCP/IP is usually disk I/O speed on either end. 99% of the time someone screws with an MTU it breaks something.
You should be ready to start writing network applications now

[edit]
My math was off. The total TCP header size can be up to 40 bytes, not just 32. That is insignificant in the points made though.
[/edit]
Last edited by sknake; Oct 10th, 2009 at 10:00 pm.
![]() |
Similar Threads
- How to synchronize a file upload with an insert in MySQL through Javascript-AJAX-PHP? (JavaScript / DHTML / AJAX)
- Sending files in socket problem/// (Java)
- sending files (Java)
- how to embed sound files in xml files? (C#)
- Sockets - listening and sending at the same time (TCP) (C++)
- Sending ByteBuffer on TCP and UDP Connection (Java)
- sending files over LAN thru C programs (C++)
- FTP-ing never felt so good (Web Browsers)
- hijack-log...thank for any help (Viruses, Spyware and other Nasties)
Other Threads in the C# Forum
- Previous Thread: msi silent install?
- Next Thread: script to run on the server
| Thread Tools | Search this Thread |
api bluetooth browsing bugs business community creepy declarations development events facebook facelift firefox google government hardware ibdg infrastructure internationalbusinessdevelopmentgroup internet investors jobs linkedin mapping marketing media month mozilla myspace network networking news php politics port protocol provider routers science sms social socialnetwork socket sockets tcp technology twitter vb.net warning web







