| | |
sending files over TCP
Please support our C# advertiser: Intel Parallel Studio Home
Thread Solved |
0
#11 Oct 10th, 2009
wow, thanks sknake, i decided on 1024 bytes for my file packet read and writes. I seem to get ok results with it. I am going to spend a few reviewing your code and see what I can make sense of it. It appears you are using remoting, but its hard to tell, since I am new to network programming.
I just figured out a simple way to loop through the bytes of a file on disk and send it out with loop, and to read it with a similar loop. the trick was to send the length of the file in bytes to the server first. what I ended up doing was sending "#FILE:numerofbytes" to the server, when a message was received from the client it checks if it starts with a "#", if so it passes the message into my command processing method, where it checks if it starts with "#FILE" if so it splits it at the ":" and parses the numbers at the end into a long int and then I use that information to start a loop that first sends a message saying its ok to send the file, ("#RFFS") then starts the read, the client then gets the message and starts the loop of sending the file in 1024 byte chunks. the client loops and sends, the server loops and revives, part of the loop writes the bytes as they are received to a file stream its really choppy but it works, and allows for an easy durring loop calculation of percentage sent/recived on both sides.
Question1: - Is my solution an acceptable one?
or should I really use the asynchronous begin and end send file methods of the socket class? (I had trouble figuring them out)
I just figured out a simple way to loop through the bytes of a file on disk and send it out with loop, and to read it with a similar loop. the trick was to send the length of the file in bytes to the server first. what I ended up doing was sending "#FILE:numerofbytes" to the server, when a message was received from the client it checks if it starts with a "#", if so it passes the message into my command processing method, where it checks if it starts with "#FILE" if so it splits it at the ":" and parses the numbers at the end into a long int and then I use that information to start a loop that first sends a message saying its ok to send the file, ("#RFFS") then starts the read, the client then gets the message and starts the loop of sending the file in 1024 byte chunks. the client loops and sends, the server loops and revives, part of the loop writes the bytes as they are received to a file stream its really choppy but it works, and allows for an easy durring loop calculation of percentage sent/recived on both sides.
Question1: - Is my solution an acceptable one?
or should I really use the asynchronous begin and end send file methods of the socket class? (I had trouble figuring them out)
0
#12 Oct 10th, 2009
No that code doesn't use remoting, just some of the messaging functionality for callbacks.
You should probably learn Asynch sockets since you don't want to block your main thread while it is processing and Async vs sync is a different code model in the back end so if you build on top of it you may have to refactor more code.
Take a look at the code I posted and see if it makes sense.
You should probably learn Asynch sockets since you don't want to block your main thread while it is processing and Async vs sync is a different code model in the back end so if you build on top of it you may have to refactor more code.
Take a look at the code I posted and see if it makes sense.
1
#13 Oct 10th, 2009
Oh also -- the code I posted is all asynchronous so you can send files concurrently
And it will have three file transmissions occuring concurrently on three separate sockets. It will fire off events as each file xfer completes.
C# Syntax (Toggle Plain Text)
client.SendFile("1"); client.SendFile("2"); client.SendFile("3");
And it will have three file transmissions occuring concurrently on three separate sockets. It will fire off events as each file xfer completes.
Last edited by sknake; Oct 10th, 2009 at 11:16 pm.
0
#14 Oct 10th, 2009
I loaded up your example solution, ran the apps and it works, I can say that
. looked through the code, and it looks beautiful. Much better than my chicken scratch. I understand most of it. I really like the custom event argument approach. Its a lot of code to take in. I am not up on threading, I understand how to create threads, and I understand calling methods on the main thread from another thread needs to be invoked... other than that. yeah... so the terms like callback throw me off, and the methods formed like this one:
that "new func<socket, byte[]>" line I have never seen before.
the client code seems straight forward, but if you get some free time and feel like it. I sure would like a little more explination on the server code. But only if you have time. Thanks alot.
I tested my 2 little loops on a 409mb file, and it works great on a local network. Im about to test yours with the same file and see how well it works. Thanks again, you are awesome!
. looked through the code, and it looks beautiful. Much better than my chicken scratch. I understand most of it. I really like the custom event argument approach. Its a lot of code to take in. I am not up on threading, I understand how to create threads, and I understand calling methods on the main thread from another thread needs to be invoked... other than that. yeah... so the terms like callback throw me off, and the methods formed like this one: C# Syntax (Toggle Plain Text)
private void ConnectionCallback(IAsyncResult ar) { TcpListener listener = (TcpListener)ar.AsyncState; try { Socket s = listener.EndAcceptSocket(ar); new Func<Socket, byte[]>(HandleSocketComms).BeginInvoke( s, new AsyncCallback(HandleSocketCommsCallback), s); } catch { //You should handle this but this should be a _rare_ error throw; } finally { //Prime up the listener to accept another inbound request listener.BeginAcceptSocket(new AsyncCallback(ConnectionCallback), listener); } }
that "new func<socket, byte[]>" line I have never seen before.
the client code seems straight forward, but if you get some free time and feel like it. I sure would like a little more explination on the server code. But only if you have time. Thanks alot.
I tested my 2 little loops on a 409mb file, and it works great on a local network. Im about to test yours with the same file and see how well it works. Thanks again, you are awesome!
0
#15 Oct 10th, 2009
your client said my 409mb video file was sent successfully in 50 seconds. but the server side windows first ran out of virtual memory and increased its default paging size. not sure why because that system has 2 gigs of ram) anyhow. then when your app auto open windows media player to play the video file, (I modified the file extention form bmp to the .avi ext) windows media player gave an error message box with soemthing in hex) and exited. I tried to modify your server to use a filestream instead of a memory stream but I was unsuccessful. as exactly what that memory stream was doing is unclear to me.
0
#16 Oct 10th, 2009
Sorry, Im back. I understand that the memory stream is being returned, I just don't see where its getting sent to, I understand from the event handler code on the form that the memory stream somehow gets sent as an event object, I just don't see how. I am sure I will eventually figure it out. maybe after a break. i have been either reading about sockets, or mashing up socket code for a good 7 hours now and i think after a good break it will make more sense. I appreciate it sknake. you are a great asset to this site!, (you too danny, thanks again for the pdf);
0
#17 Oct 11th, 2009
Visual studio's intelisense gave me all the information I needed to know to change the return type of the HandleSocketComms method and change the the memory stream to a file stream, it all works well except using this method, I can't seem to figure out how to send the file extension from the client to the server.
I think I can extend these classes you have provided to do great things for my applications. I can't say it enough. I appreciate the help.
I think I can extend these classes you have provided to do great things for my applications. I can't say it enough. I appreciate the help.
0
#18 Oct 11th, 2009
Yes -- I did know it wouldn't work for large files that is why i put:
For large files your solution is to change that memorystream to a file stream. Use
The application I gave you was not intended to handle sending the file information over the socket, just to send the raw data asynchronously. But with a little work you could implement a command structure and a file structure on the same socket.
If you have a specific question about any of the comms let me know and maybe we can get your problem sorted out
C# Syntax (Toggle Plain Text)
using (MemoryStream ms = new MemoryStream()) //This may be a bad idea for big files :)
For large files your solution is to change that memorystream to a file stream. Use
Path.GetTempFile() and start piping the contents to a file instead of memory. You would also want to change the event to pass the string of the temp file name instead of the byte array, and then handle the event as a file.The application I gave you was not intended to handle sending the file information over the socket, just to send the raw data asynchronously. But with a little work you could implement a command structure and a file structure on the same socket.
If you have a specific question about any of the comms let me know and maybe we can get your problem sorted out
first off I modified your custom arguments to take a string
then I modified your connection callback method to return a string instead of a byte array.
then I modified your handlesocketcomms method to save to a file stream
Finally I modified your handlesocketcommscallback method to get the new eventarg and create the new eventargument class
now I want to be straight forward with you. I understand the flow of code here. I see the order of code execution, and I understand the general idea. But as for the specifics. I really have no understanding of how the IAsyncresult object works, or the asynccallback stuff.
An issue for me, is I can't figure out in this scenario how I would send information like file name, extention, and length, that way I could calculate progress. In the one I did I sent text across the stream and waited for an answer on the client before I started sending the information. but with your code, I am just kinda confused with where to do this.
I do appreciate all the help. ( the file stream version did successfully transmit the 409mb file across my home network)
C# Syntax (Toggle Plain Text)
public class OnFileReceivedArgs : EventArgs { private byte[] _buffer; public byte[] Buffer { get { return _buffer; } } private string _FilePath; public string FilePath { get { return _FilePath; } } private OnFileReceivedArgs() : base() { } public OnFileReceivedArgs(byte[] buffer) : this() { this._buffer = buffer; } public OnFileReceivedArgs(string filePath) : this() { this._FilePath = filePath; } }
then I modified your connection callback method to return a string instead of a byte array.
C# Syntax (Toggle Plain Text)
private void ConnectionCallback(IAsyncResult ar) { TcpListener listener = (TcpListener)ar.AsyncState; try { Socket s = listener.EndAcceptSocket(ar); new Func<Socket, string>(HandleSocketComms).BeginInvoke( s, new AsyncCallback(HandleSocketCommsCallback), s); } catch { //You should handle this but this should be a _rare_ error throw; } finally { //Prime up the listener to accept another inbound request listener.BeginAcceptSocket(new AsyncCallback(ConnectionCallback), listener); } }
then I modified your handlesocketcomms method to save to a file stream
C# Syntax (Toggle Plain Text)
private string HandleSocketComms(Socket s) { //Dont do any exception handling here TrackThread(); string tempfileloc = Path.GetTempFileName(); using (FileStream ms = new FileStream(tempfileloc, FileMode.Create, FileAccess.Write)) //This may be a bad idea for big files :) { do { int szToRead = s.Available; byte[] buffer = new byte[szToRead]; int szRead = s.Receive(buffer, szToRead, SocketFlags.None); if (szRead > 0) { ms.Write(buffer, 0, szRead); Console.WriteLine("Read " + szRead.ToString()); } } while (SocketConnected(s)); return tempfileloc; } //return result; }
Finally I modified your handlesocketcommscallback method to get the new eventarg and create the new eventargument class
C# Syntax (Toggle Plain Text)
private void HandleSocketCommsCallback(IAsyncResult ar) { byte[] file = null; string filepath = ""; bool error = false; try { AsyncResult result = (AsyncResult)ar; var del = (Func<Socket, string>)result.AsyncDelegate; filepath = del.EndInvoke(ar); Socket s = (ar.AsyncState as Socket); if (s != null) { //Sockets throw exceptions++ when disposed try { s.Close(0); ((IDisposable)s).Dispose(); } catch { } } } catch (ThreadAbortException) //When we stop our server { //We were killed! Just let it die out error = true; } catch { //Might want to handle this error = true; } finally { RemoveThread(); //conn is closed at this point //if (!error && (file != null) && (file.Length > 0)) if (!error) { var del = this.OnFileReceived; if (del != null) del(this, new OnFileReceivedArgs(filepath)); } } }
now I want to be straight forward with you. I understand the flow of code here. I see the order of code execution, and I understand the general idea. But as for the specifics. I really have no understanding of how the IAsyncresult object works, or the asynccallback stuff.
An issue for me, is I can't figure out in this scenario how I would send information like file name, extention, and length, that way I could calculate progress. In the one I did I sent text across the stream and waited for an answer on the client before I started sending the information. but with your code, I am just kinda confused with where to do this.
I do appreciate all the help. ( the file stream version did successfully transmit the 409mb file across my home network)
Last edited by Diamonddrake; Oct 11th, 2009 at 2:34 pm. Reason: missed a set of code brackets
0
#20 Oct 11th, 2009
OK this is going back to my original comments about how FTP does.
Solution 1)
(client->server) I'm sending blah.txt 4096 bytes
(server->client) ok hit me up on port:30044 for the xfer
(client->server) connects and sends file
Solution 2)
(client->server) I'm sending blah.txt 4096 bytes
(client->server) sends file over existing connection
Solution 1 requires two ports (= more firewall configurations)
Solution 2 requires more overhead for inspecting the data
From our conversation I feel that you want to use solution 2, so:
I would create another assembly ClientServer.Common as a class library and define a class like:
Now you can't have undisplayable characters and line feeds inside the body of an XML document or it will blow up the DOM parser in the .NET framework so our signature is safe in that regard for delimiting between a command and file. You would create the signature and send a packet like:
client->server
(command.Serialize()) + COMMAND_DELIM + FILE_DATA
What the server would do is this (you will have to undo the work you previously did):
In the HandleSocksComms it would need to buffer the inbound data in a memory stream again but this time we won't let it run the system out of memory.
As data is received it needs to add the buffer to the memory stream
After the data is appending it needs to check to see if the signature is found
> If the signature is found then send all byte[]s before the signature to static methods of the command class to parse the serialized data
> If the signature is not found then continue buffering the input
> If the buffered input is greater than 10KB then kill the socket because someone has connected to you and is just sending you crap. The serialized command class as it stands will be very small
So to visually represent the packet you could see it like this
C=Command, D=Delimiter, F=File
CCCCCCDDDDDDFFFFFFFFFFFFFFF
You can't check the indivual buffer you just received for the signature because it is possible that the packets will fragment across signature, like so:
PACKET1 | PACKET2
CCCCCCDDD | DDDFFFFFFFFFFFFFFF
Now if you fail to deserialize the class then kill the socket because its some unknown software on the other end, again, sending crap.
What you could do is look at the XML DOM's specs and see if there is any single character that is not allow in XML (I know there is) and use that single byte as your delimiter. IE if you can't have a null (=0 value) then you could separate them like:
CCCCC0FFFFF
And since a single byte cannot be fragmented across the sends you wouldn't have to check the in-memory buffer and life will be easier separating between commands and files.
Now once you have the command deserialized in memory you will read the stream the exact length of the file as indicated in the Command's FileSize property. If the buffer is smaller then throw it all away, the transmission did not complete. After you have read the exact length of the file you will be in "waiting for command" mode again where you will continue to read until you hit a sig and attempt to deserialize the signature.
You cannot rely on a command delimiter after the binary file because say your deserialized command class says the file size is 4000 bytes, but the client actually sends 4500 bytes of data -- it could have whatever delimiter you specify in the file because it is binary data and any byte value is potentially valid. Example
CCCCCDDDFFFFFFFFFDDDFFFF
In this case the file, as a matter of luck, just so happens to contain your signature and your application would break when someone sends this file. This would lead to a very hard to diagnose issue.
[disclaimer]
Now -- All of these concepts above are perfectly valid and will work but they may not be the most efficient. They are very good for academic purposes or even an application with low-moderate usage. If thousands of users are going to use this app then every bit matters and this would be seen as a horribly ineffecient waste of space, bandwidth, memory, and disk space. If we really wanted to be high performance we would use less than byte to transmit the command and mask the bits out on either end to keep storage space to a minimal but this gets a little complicated. Plus we wouldn't use the XmlSerializer because that is a little slow...
[/disclaimer]
Solution 1)
(client->server) I'm sending blah.txt 4096 bytes
(server->client) ok hit me up on port:30044 for the xfer
(client->server) connects and sends file
Solution 2)
(client->server) I'm sending blah.txt 4096 bytes
(client->server) sends file over existing connection
Solution 1 requires two ports (= more firewall configurations)
Solution 2 requires more overhead for inspecting the data
From our conversation I feel that you want to use solution 2, so:
I would create another assembly ClientServer.Common as a class library and define a class like:
C# Syntax (Toggle Plain Text)
using System; using System.Collections.Generic; using System.Linq; using System.Text; using System.IO; using System.Xml.Serialization; namespace ClientServer.Common { public enum CommandType { SendFile } public class Command { /// <summary> /// When this byte collection is hit you know you have received a command /// </summary> public static readonly byte[] COMMAND_DELIM = new byte[] { 7, 9, 13, 10, 10, 13 }; //This is invalid inside an XML document so its "good-enough" public CommandType CommandCode { get; set; } /// <summary> /// File size in bytes /// </summary> public int FileSize { get; set; } public string FileName { get; set; } public Command() { } public byte[] Serialize() { using (MemoryStream ms = new MemoryStream()) { XmlSerializer ser = new XmlSerializer(typeof(Command)); ser.Serialize(ms, this); return ms.ToArray(); } } public static Command FromBuffer(byte[] buffer) { using (MemoryStream ms = new MemoryStream(buffer)) { return FromStream(ms); } } public static Command FromStream(Stream s) { s.Seek(0, SeekOrigin.Begin); s.Position = 0; XmlSerializer ser = new XmlSerializer(typeof(Command)); return (Command)ser.Deserialize(s); } } }
Now you can't have undisplayable characters and line feeds inside the body of an XML document or it will blow up the DOM parser in the .NET framework so our signature is safe in that regard for delimiting between a command and file. You would create the signature and send a packet like:
client->server
(command.Serialize()) + COMMAND_DELIM + FILE_DATA
What the server would do is this (you will have to undo the work you previously did):
In the HandleSocksComms it would need to buffer the inbound data in a memory stream again but this time we won't let it run the system out of memory.
As data is received it needs to add the buffer to the memory stream
After the data is appending it needs to check to see if the signature is found
> If the signature is found then send all byte[]s before the signature to static methods of the command class to parse the serialized data
> If the signature is not found then continue buffering the input
> If the buffered input is greater than 10KB then kill the socket because someone has connected to you and is just sending you crap. The serialized command class as it stands will be very small
So to visually represent the packet you could see it like this
C=Command, D=Delimiter, F=File
CCCCCCDDDDDDFFFFFFFFFFFFFFF
You can't check the indivual buffer you just received for the signature because it is possible that the packets will fragment across signature, like so:
PACKET1 | PACKET2
CCCCCCDDD | DDDFFFFFFFFFFFFFFF
Now if you fail to deserialize the class then kill the socket because its some unknown software on the other end, again, sending crap.
What you could do is look at the XML DOM's specs and see if there is any single character that is not allow in XML (I know there is) and use that single byte as your delimiter. IE if you can't have a null (=0 value) then you could separate them like:
CCCCC0FFFFF
And since a single byte cannot be fragmented across the sends you wouldn't have to check the in-memory buffer and life will be easier separating between commands and files.
Now once you have the command deserialized in memory you will read the stream the exact length of the file as indicated in the Command's FileSize property. If the buffer is smaller then throw it all away, the transmission did not complete. After you have read the exact length of the file you will be in "waiting for command" mode again where you will continue to read until you hit a sig and attempt to deserialize the signature.
You cannot rely on a command delimiter after the binary file because say your deserialized command class says the file size is 4000 bytes, but the client actually sends 4500 bytes of data -- it could have whatever delimiter you specify in the file because it is binary data and any byte value is potentially valid. Example
CCCCCDDDFFFFFFFFFDDDFFFF
In this case the file, as a matter of luck, just so happens to contain your signature and your application would break when someone sends this file. This would lead to a very hard to diagnose issue.
[disclaimer]
Now -- All of these concepts above are perfectly valid and will work but they may not be the most efficient. They are very good for academic purposes or even an application with low-moderate usage. If thousands of users are going to use this app then every bit matters and this would be seen as a horribly ineffecient waste of space, bandwidth, memory, and disk space. If we really wanted to be high performance we would use less than byte to transmit the command and mask the bits out on either end to keep storage space to a minimal but this gets a little complicated. Plus we wouldn't use the XmlSerializer because that is a little slow...
[/disclaimer]
![]() |
Similar Threads
- How to synchronize a file upload with an insert in MySQL through Javascript-AJAX-PHP? (JavaScript / DHTML / AJAX)
- Sending files in socket problem/// (Java)
- sending files (Java)
- how to embed sound files in xml files? (C#)
- Sockets - listening and sending at the same time (TCP) (C++)
- Sending ByteBuffer on TCP and UDP Connection (Java)
- sending files over LAN thru C programs (C++)
- FTP-ing never felt so good (Web Browsers)
- hijack-log...thank for any help (Viruses, Spyware and other Nasties)
Other Threads in the C# Forum
- Previous Thread: msi silent install?
- Next Thread: script to run on the server
| Thread Tools | Search this Thread |
api bluetooth books browsing bugs business community creepy declarations development dns events facebook facelift firefox google government hardware ibdg infrastructure internationalbusinessdevelopmentgroup internet investors ip jobs linkedin listener mapping marketing media month mozilla myspace network networking news php politics port programming protocol provider routers science security sms social socialnetwork socket sockets software tcp technology twitter vb.net warning web







