sending files over TCP

Please support our C# advertiser: Intel Parallel Studio Home
Thread Solved

Join Date: Mar 2008
Posts: 330
Reputation: Diamonddrake is a jewel in the rough Diamonddrake is a jewel in the rough Diamonddrake is a jewel in the rough 
Solved Threads: 39
Diamonddrake's Avatar
Diamonddrake Diamonddrake is offline Offline
Posting Whiz
 
0
  #11
Oct 10th, 2009
wow, thanks sknake, i decided on 1024 bytes for my file packet read and writes. I seem to get ok results with it. I am going to spend a few reviewing your code and see what I can make sense of it. It appears you are using remoting, but its hard to tell, since I am new to network programming.

I just figured out a simple way to loop through the bytes of a file on disk and send it out with loop, and to read it with a similar loop. the trick was to send the length of the file in bytes to the server first. what I ended up doing was sending "#FILE:numerofbytes" to the server, when a message was received from the client it checks if it starts with a "#", if so it passes the message into my command processing method, where it checks if it starts with "#FILE" if so it splits it at the ":" and parses the numbers at the end into a long int and then I use that information to start a loop that first sends a message saying its ok to send the file, ("#RFFS") then starts the read, the client then gets the message and starts the loop of sending the file in 1024 byte chunks. the client loops and sends, the server loops and revives, part of the loop writes the bytes as they are received to a file stream its really choppy but it works, and allows for an easy durring loop calculation of percentage sent/recived on both sides.

Question1: - Is my solution an acceptable one?
or should I really use the asynchronous begin and end send file methods of the socket class? (I had trouble figuring them out)
Reply With Quote Quick reply to this message  
Join Date: Feb 2009
Posts: 3,230
Reputation: sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of 
Solved Threads: 576
Sponsor
sknake's Avatar
sknake sknake is offline Offline
.NET Enthusiast
 
0
  #12
Oct 10th, 2009
No that code doesn't use remoting, just some of the messaging functionality for callbacks.

You should probably learn Asynch sockets since you don't want to block your main thread while it is processing and Async vs sync is a different code model in the back end so if you build on top of it you may have to refactor more code.

Take a look at the code I posted and see if it makes sense.
Scott Knake
Custom Software Development
Apex Software, Inc.
Reply With Quote Quick reply to this message  
Join Date: Feb 2009
Posts: 3,230
Reputation: sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of 
Solved Threads: 576
Sponsor
sknake's Avatar
sknake sknake is offline Offline
.NET Enthusiast
 
1
  #13
Oct 10th, 2009
Oh also -- the code I posted is all asynchronous so you can send files concurrently

  1. client.SendFile("1");
  2. client.SendFile("2");
  3. client.SendFile("3");

And it will have three file transmissions occuring concurrently on three separate sockets. It will fire off events as each file xfer completes.
Last edited by sknake; Oct 10th, 2009 at 11:16 pm.
Scott Knake
Custom Software Development
Apex Software, Inc.
Reply With Quote Quick reply to this message  
Join Date: Mar 2008
Posts: 330
Reputation: Diamonddrake is a jewel in the rough Diamonddrake is a jewel in the rough Diamonddrake is a jewel in the rough 
Solved Threads: 39
Diamonddrake's Avatar
Diamonddrake Diamonddrake is offline Offline
Posting Whiz
 
0
  #14
Oct 10th, 2009
I loaded up your example solution, ran the apps and it works, I can say that . looked through the code, and it looks beautiful. Much better than my chicken scratch. I understand most of it. I really like the custom event argument approach. Its a lot of code to take in. I am not up on threading, I understand how to create threads, and I understand calling methods on the main thread from another thread needs to be invoked... other than that. yeah... so the terms like callback throw me off, and the methods formed like this one:

  1. private void ConnectionCallback(IAsyncResult ar)
  2. {
  3. TcpListener listener = (TcpListener)ar.AsyncState;
  4. try
  5. {
  6. Socket s = listener.EndAcceptSocket(ar);
  7. new Func<Socket, byte[]>(HandleSocketComms).BeginInvoke(
  8. s,
  9. new AsyncCallback(HandleSocketCommsCallback),
  10. s);
  11. }
  12. catch
  13. {
  14. //You should handle this but this should be a _rare_ error
  15. throw;
  16. }
  17. finally
  18. {
  19. //Prime up the listener to accept another inbound request
  20. listener.BeginAcceptSocket(new AsyncCallback(ConnectionCallback), listener);
  21. }
  22. }

that "new func<socket, byte[]>" line I have never seen before.

the client code seems straight forward, but if you get some free time and feel like it. I sure would like a little more explination on the server code. But only if you have time. Thanks alot.

I tested my 2 little loops on a 409mb file, and it works great on a local network. Im about to test yours with the same file and see how well it works. Thanks again, you are awesome!
Reply With Quote Quick reply to this message  
Join Date: Mar 2008
Posts: 330
Reputation: Diamonddrake is a jewel in the rough Diamonddrake is a jewel in the rough Diamonddrake is a jewel in the rough 
Solved Threads: 39
Diamonddrake's Avatar
Diamonddrake Diamonddrake is offline Offline
Posting Whiz
 
0
  #15
Oct 10th, 2009
your client said my 409mb video file was sent successfully in 50 seconds. but the server side windows first ran out of virtual memory and increased its default paging size. not sure why because that system has 2 gigs of ram) anyhow. then when your app auto open windows media player to play the video file, (I modified the file extention form bmp to the .avi ext) windows media player gave an error message box with soemthing in hex) and exited. I tried to modify your server to use a filestream instead of a memory stream but I was unsuccessful. as exactly what that memory stream was doing is unclear to me.
Reply With Quote Quick reply to this message  
Join Date: Mar 2008
Posts: 330
Reputation: Diamonddrake is a jewel in the rough Diamonddrake is a jewel in the rough Diamonddrake is a jewel in the rough 
Solved Threads: 39
Diamonddrake's Avatar
Diamonddrake Diamonddrake is offline Offline
Posting Whiz
 
0
  #16
Oct 10th, 2009
Sorry, Im back. I understand that the memory stream is being returned, I just don't see where its getting sent to, I understand from the event handler code on the form that the memory stream somehow gets sent as an event object, I just don't see how. I am sure I will eventually figure it out. maybe after a break. i have been either reading about sockets, or mashing up socket code for a good 7 hours now and i think after a good break it will make more sense. I appreciate it sknake. you are a great asset to this site!, (you too danny, thanks again for the pdf);
Reply With Quote Quick reply to this message  
Join Date: Mar 2008
Posts: 330
Reputation: Diamonddrake is a jewel in the rough Diamonddrake is a jewel in the rough Diamonddrake is a jewel in the rough 
Solved Threads: 39
Diamonddrake's Avatar
Diamonddrake Diamonddrake is offline Offline
Posting Whiz
 
0
  #17
Oct 11th, 2009
Visual studio's intelisense gave me all the information I needed to know to change the return type of the HandleSocketComms method and change the the memory stream to a file stream, it all works well except using this method, I can't seem to figure out how to send the file extension from the client to the server.

I think I can extend these classes you have provided to do great things for my applications. I can't say it enough. I appreciate the help.
Reply With Quote Quick reply to this message  
Join Date: Feb 2009
Posts: 3,230
Reputation: sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of 
Solved Threads: 576
Sponsor
sknake's Avatar
sknake sknake is offline Offline
.NET Enthusiast
 
0
  #18
Oct 11th, 2009
Yes -- I did know it wouldn't work for large files that is why i put:
  1. using (MemoryStream ms = new MemoryStream()) //This may be a bad idea for big files :)

For large files your solution is to change that memorystream to a file stream. Use Path.GetTempFile() and start piping the contents to a file instead of memory. You would also want to change the event to pass the string of the temp file name instead of the byte array, and then handle the event as a file.

The application I gave you was not intended to handle sending the file information over the socket, just to send the raw data asynchronously. But with a little work you could implement a command structure and a file structure on the same socket.

If you have a specific question about any of the comms let me know and maybe we can get your problem sorted out
Scott Knake
Custom Software Development
Apex Software, Inc.
Reply With Quote Quick reply to this message  
Join Date: Mar 2008
Posts: 330
Reputation: Diamonddrake is a jewel in the rough Diamonddrake is a jewel in the rough Diamonddrake is a jewel in the rough 
Solved Threads: 39
Diamonddrake's Avatar
Diamonddrake Diamonddrake is offline Offline
Posting Whiz

modifications.

 
0
  #19
Oct 11th, 2009
first off I modified your custom arguments to take a string

  1. public class OnFileReceivedArgs : EventArgs
  2. {
  3. private byte[] _buffer;
  4. public byte[] Buffer { get { return _buffer; } }
  5.  
  6. private string _FilePath;
  7. public string FilePath { get { return _FilePath; } }
  8.  
  9. private OnFileReceivedArgs()
  10. : base()
  11. {
  12. }
  13. public OnFileReceivedArgs(byte[] buffer)
  14. : this()
  15. {
  16. this._buffer = buffer;
  17. }
  18.  
  19. public OnFileReceivedArgs(string filePath)
  20. : this()
  21. {
  22. this._FilePath = filePath;
  23. }
  24. }

then I modified your connection callback method to return a string instead of a byte array.

  1. private void ConnectionCallback(IAsyncResult ar)
  2. {
  3. TcpListener listener = (TcpListener)ar.AsyncState;
  4. try
  5. {
  6. Socket s = listener.EndAcceptSocket(ar);
  7. new Func<Socket, string>(HandleSocketComms).BeginInvoke(
  8. s,
  9. new AsyncCallback(HandleSocketCommsCallback),
  10. s);
  11.  
  12.  
  13. }
  14. catch
  15. {
  16. //You should handle this but this should be a _rare_ error
  17. throw;
  18. }
  19. finally
  20. {
  21. //Prime up the listener to accept another inbound request
  22. listener.BeginAcceptSocket(new AsyncCallback(ConnectionCallback), listener);
  23. }
  24. }

then I modified your handlesocketcomms method to save to a file stream

  1. private string HandleSocketComms(Socket s)
  2. {
  3. //Dont do any exception handling here
  4. TrackThread();
  5. string tempfileloc = Path.GetTempFileName();
  6. using (FileStream ms = new FileStream(tempfileloc, FileMode.Create, FileAccess.Write)) //This may be a bad idea for big files :)
  7. {
  8. do
  9. {
  10. int szToRead = s.Available;
  11. byte[] buffer = new byte[szToRead];
  12.  
  13. int szRead = s.Receive(buffer, szToRead, SocketFlags.None);
  14. if (szRead > 0)
  15. {
  16. ms.Write(buffer, 0, szRead);
  17. Console.WriteLine("Read " + szRead.ToString());
  18. }
  19. }
  20. while (SocketConnected(s));
  21.  
  22. return tempfileloc;
  23. }
  24. //return result;
  25. }

Finally I modified your handlesocketcommscallback method to get the new eventarg and create the new eventargument class

  1. private void HandleSocketCommsCallback(IAsyncResult ar)
  2. {
  3. byte[] file = null;
  4. string filepath = "";
  5. bool error = false;
  6.  
  7. try
  8. {
  9. AsyncResult result = (AsyncResult)ar;
  10. var del = (Func<Socket, string>)result.AsyncDelegate;
  11. filepath = del.EndInvoke(ar);
  12.  
  13. Socket s = (ar.AsyncState as Socket);
  14. if (s != null)
  15. {
  16. //Sockets throw exceptions++ when disposed
  17. try
  18. {
  19. s.Close(0);
  20. ((IDisposable)s).Dispose();
  21. }
  22. catch { }
  23. }
  24. }
  25. catch (ThreadAbortException) //When we stop our server
  26. {
  27. //We were killed! Just let it die out
  28. error = true;
  29. }
  30. catch
  31. {
  32. //Might want to handle this
  33. error = true;
  34. }
  35. finally
  36. {
  37. RemoveThread(); //conn is closed at this point
  38. //if (!error && (file != null) && (file.Length > 0))
  39. if (!error)
  40. {
  41. var del = this.OnFileReceived;
  42. if (del != null)
  43. del(this, new OnFileReceivedArgs(filepath));
  44. }
  45. }
  46. }

now I want to be straight forward with you. I understand the flow of code here. I see the order of code execution, and I understand the general idea. But as for the specifics. I really have no understanding of how the IAsyncresult object works, or the asynccallback stuff.

An issue for me, is I can't figure out in this scenario how I would send information like file name, extention, and length, that way I could calculate progress. In the one I did I sent text across the stream and waited for an answer on the client before I started sending the information. but with your code, I am just kinda confused with where to do this.

I do appreciate all the help. ( the file stream version did successfully transmit the 409mb file across my home network)
Last edited by Diamonddrake; Oct 11th, 2009 at 2:34 pm. Reason: missed a set of code brackets
Reply With Quote Quick reply to this message  
Join Date: Feb 2009
Posts: 3,230
Reputation: sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of sknake has much to be proud of 
Solved Threads: 576
Sponsor
sknake's Avatar
sknake sknake is offline Offline
.NET Enthusiast
 
0
  #20
Oct 11th, 2009
OK this is going back to my original comments about how FTP does.

Solution 1)
(client->server) I'm sending blah.txt 4096 bytes
(server->client) ok hit me up on port:30044 for the xfer
(client->server) connects and sends file

Solution 2)
(client->server) I'm sending blah.txt 4096 bytes
(client->server) sends file over existing connection

Solution 1 requires two ports (= more firewall configurations)
Solution 2 requires more overhead for inspecting the data

From our conversation I feel that you want to use solution 2, so:
I would create another assembly ClientServer.Common as a class library and define a class like:

  1. using System;
  2. using System.Collections.Generic;
  3. using System.Linq;
  4. using System.Text;
  5. using System.IO;
  6. using System.Xml.Serialization;
  7.  
  8. namespace ClientServer.Common
  9. {
  10. public enum CommandType
  11. {
  12. SendFile
  13. }
  14. public class Command
  15. {
  16.  
  17. /// <summary>
  18. /// When this byte collection is hit you know you have received a command
  19. /// </summary>
  20. public static readonly byte[] COMMAND_DELIM = new byte[] { 7, 9, 13, 10, 10, 13 }; //This is invalid inside an XML document so its "good-enough"
  21.  
  22.  
  23. public CommandType CommandCode { get; set; }
  24. /// <summary>
  25. /// File size in bytes
  26. /// </summary>
  27. public int FileSize { get; set; }
  28. public string FileName { get; set; }
  29. public Command()
  30. {
  31. }
  32.  
  33. public byte[] Serialize()
  34. {
  35. using (MemoryStream ms = new MemoryStream())
  36. {
  37. XmlSerializer ser = new XmlSerializer(typeof(Command));
  38. ser.Serialize(ms, this);
  39. return ms.ToArray();
  40. }
  41. }
  42.  
  43. public static Command FromBuffer(byte[] buffer)
  44. {
  45. using (MemoryStream ms = new MemoryStream(buffer))
  46. {
  47. return FromStream(ms);
  48. }
  49. }
  50. public static Command FromStream(Stream s)
  51. {
  52. s.Seek(0, SeekOrigin.Begin);
  53. s.Position = 0;
  54. XmlSerializer ser = new XmlSerializer(typeof(Command));
  55. return (Command)ser.Deserialize(s);
  56. }
  57. }
  58. }

Now you can't have undisplayable characters and line feeds inside the body of an XML document or it will blow up the DOM parser in the .NET framework so our signature is safe in that regard for delimiting between a command and file. You would create the signature and send a packet like:

client->server
(command.Serialize()) + COMMAND_DELIM + FILE_DATA

What the server would do is this (you will have to undo the work you previously did):
In the HandleSocksComms it would need to buffer the inbound data in a memory stream again but this time we won't let it run the system out of memory.

As data is received it needs to add the buffer to the memory stream
After the data is appending it needs to check to see if the signature is found
> If the signature is found then send all byte[]s before the signature to static methods of the command class to parse the serialized data
> If the signature is not found then continue buffering the input
> If the buffered input is greater than 10KB then kill the socket because someone has connected to you and is just sending you crap. The serialized command class as it stands will be very small

So to visually represent the packet you could see it like this
C=Command, D=Delimiter, F=File
CCCCCCDDDDDDFFFFFFFFFFFFFFF

You can't check the indivual buffer you just received for the signature because it is possible that the packets will fragment across signature, like so:
PACKET1 | PACKET2
CCCCCCDDD | DDDFFFFFFFFFFFFFFF

Now if you fail to deserialize the class then kill the socket because its some unknown software on the other end, again, sending crap.

What you could do is look at the XML DOM's specs and see if there is any single character that is not allow in XML (I know there is) and use that single byte as your delimiter. IE if you can't have a null (=0 value) then you could separate them like:
CCCCC0FFFFF

And since a single byte cannot be fragmented across the sends you wouldn't have to check the in-memory buffer and life will be easier separating between commands and files.

Now once you have the command deserialized in memory you will read the stream the exact length of the file as indicated in the Command's FileSize property. If the buffer is smaller then throw it all away, the transmission did not complete. After you have read the exact length of the file you will be in "waiting for command" mode again where you will continue to read until you hit a sig and attempt to deserialize the signature.

You cannot rely on a command delimiter after the binary file because say your deserialized command class says the file size is 4000 bytes, but the client actually sends 4500 bytes of data -- it could have whatever delimiter you specify in the file because it is binary data and any byte value is potentially valid. Example
CCCCCDDDFFFFFFFFFDDDFFFF
In this case the file, as a matter of luck, just so happens to contain your signature and your application would break when someone sends this file. This would lead to a very hard to diagnose issue.

[disclaimer]
Now -- All of these concepts above are perfectly valid and will work but they may not be the most efficient. They are very good for academic purposes or even an application with low-moderate usage. If thousands of users are going to use this app then every bit matters and this would be seen as a horribly ineffecient waste of space, bandwidth, memory, and disk space. If we really wanted to be high performance we would use less than byte to transmit the command and mask the bits out on either end to keep storage space to a minimal but this gets a little complicated. Plus we wouldn't use the XmlSerializer because that is a little slow...
[/disclaimer]
Scott Knake
Custom Software Development
Apex Software, Inc.
Reply With Quote Quick reply to this message  
Reply

Tags
networking, sockets, tcp

This thread has been marked solved.
Perhaps start a new thread instead?
Message:


Thread Tools Search this Thread



About Us | Contact Us | Advertise | DaniWeb | Acceptable Use Policy | RSS Feed

©2003 - 2009 DaniWeb® LLC