Verify if data sent by servers? How do bigger companies do that?

[Aeonix]

As soon as I find out exact manner of communication between the server and the software. I realized how easy it is to use programs to intercept it, analyze it and then send malicious (or rather illegal) query through.

So. I was wondering is there a way to make sure that the query is sent by a legal program that was meant to send these queries and not a fakey.

I know it sounds obscene, these days everything can be corrupted, broken, taken apart. And interception of connection (HTTP(S) queries in this case) is everyday-bread for even the most empty-headed scriptkiddies.

But how do great companies detect/block such a thing? I was playing an online video game, it's quite popular and out there. On some day, a programmer released his very own version of the launcher (with buttons, shop, and ways to enter a game (which still would open the original "battle arena .exe")). Owners of game didn't like it. And even though program sent exactly same query as the original launcher, still, majority of users of this "new launcher" got banned for 3rd party software (as it was forbidden in ToU).

What could be the way they detected it? I mean, of course they could've updated their client to send additional token with HTTP(S) which the newly-forged program wouldn't send, and then ban everybody who didn't send that token (honey token).

But even then, the developer of his own new launcher, could've detected differences in patch version (from 4.2 => 4.3) and then block usage of it (through piece of code in launcher which would detect new version) and find out how the latest version works, update the code a bit, and keep being stealthy.

Is there really nothing that can be done? Is there other way to make it "more protected", more "legal"?

I know that server can block queries which are illegal. But I don't want someone else to create bridge to my program and then use the server to answer that one program's commands.

I'm not asking about particular programming language, let it even be Java, C++, C#, Python. Whatever. I'm not asking for code, rather for concept, I'd like to understand, if there is a better way to "protect it more".

[rubberman]

HTTPS / ssh encrypt the data stream. There are some issues with that, depending upon the hash and encryption algorithms your particular implementation is using. However, they are mostly pretty secure. As for blocking specific implementations of the game code / data stream, that is up to the implementors. They "own" it, and can allow or disallow what they want. "Legal" has nothing to do with it.

[Aeonix]

HTTPS / ssh encrypt the data stream.

Is this what you're talking about?: https://help.github.com/articles/using-ssh-over-the-https-port/
Or at least something alike? Also, does SSH over HTTPS make it somehow less interceptable? Than with pure HTTPS?
Does SSH over HTTPS work well with UDP connections? So far it seems it is 2-way encryption, wouldn't add that pretty big delay to the packet arrival? Or is there "keep-alive" connection?

I have read this https://www.digitalocean.com/community/tutorials/understanding-the-ssh-encryption-and-connection-process link about SSH (I don't know if it's correct). Isn't it how HTTPS itself works?

[overwraith]

They have some good points. You see, when you use ssh, or ssl, or some form of encryption you not only verify that no-one has modified the transmission in transit, you have what is called 'attribution' because you know that 'this transmission' has come from 'this address', and you know that it has not been tweaked in transit because you know that it is encrypted. You therefore have attribution because you know which device did what damage. Don't jump to the conclusion that the user is automatically the bad guy, take into account whether there has been malware planted on his compter, etc before you jump to conclusions, but you're forensic investigator and net admin if he they are competent and honest will be able to tell you more or less who's to blame. Many applications aren't life ending, therefore you can just block specific users.

There are also tools which can alert you to exactly what is being transmitted to your servers, network sniffers come to mind, but you have to learn all the tricks/filters associated with your specific protocol. Make sure you have permission to run one on your network.

Firewalls are good to have, and many can log suspicious activity on your network.

VLAN'ing users is also a good idea, it keeps users seperated from each other so they can't necessarily share malware if you know what I mean. It would probably play hell with whatever info sharing is supposed to occur however.

Above all, figure out where the low hanging fruit in your security setup is. Enumerate actors, vulnerabilities, and threats. etc.

Security is a group activity, and typically individuals in the company if they are aware can figure out what they need to do to make their little cubby holes more secure, but a lot of the burden for the whole scheme lies with the net admin's and security types.

There may be a diagram online which can help give you food for thought, I believe my books called it "Network Attack Life Cycle Diagram". It can give you an idea of places to look for ideas.

You should also practice information limiting, and seperation of concerns etc. Organizations like to use the phrases "need to know", and lowest possible permissions. You can think of it in a programatical sense as your public/private/protected variables etc. You want to only expose the bare minimum of variables necessary so that others can't put a stick in your spokes while your bicicle object is cruising down the road.

[Aeonix]

You see, when you use ssh, or ssl, or some form of encryption you not only verify that no-one has modified the transmission in transit,

Whoa whoa whoa! Doesn't Windows, Linux, MacOS, Android and every single operating system with internet connection, save these "passwords" somewhere which can easily be extracted and used in decryption? How would server verify that the content hasn't been modified? If I have the content and en/decryption key, I could forge new hash, couldn't I?

Don't jump to the conclusion that the user is automatically the bad guy,

...I mean no disrespect. But this is most unhealthy set-up for a programmer in my opinion. There are bad guys out there, and you should always be protected for the worst. I know that majority isn't malicious, but there's still that 1%, and I'm not going to risk not parsing data because majority isn't malicious.

There are also tools which can alert you to exactly what is being transmitted to your servers, network sniffers come to mind, but you have to learn all the tricks/filters associated with your specific protocol. Make sure you have permission to run one on your network. Firewalls are good to have, and many can log suspicious activity on your network.

Sir, you're wondering off track, I'm not asking how to protect my server against malicious string, but how could I verify if string has been sent by MY program, and not reforged. I'm assuming that the string is healthy. But attempting to execute something that is not in users allowance. For example (maybe not precisely best example), send a message to a friend, who is online, who supposedly cannot really be messaged, so button is disabled, so someone would create program, which would use query from the program, to force arrival of message (which still would be blocked, but..., y'know, I'd still like to know if my program sent it, or it has been handcrafted)

I believe my books called it "Network Attack Life Cycle Diagram".

No no no no! The question isn't about server invade, it's about verification of sent string by a program.

You want to only expose the bare minimum of variables necessary so that others can't put a stick in your spokes while your bicicle object is cruising down the road.

Yes, I do plan to parse incoming requests, but I need to verify first that the query has been sent by my program, not manually or by another automated program.

[Aeonix]

Still awaiting some more explanation.
Anyone? :)

[overwraith]

Sorry, it's 3 way encryption, there are basically 3 parties involved, and the way the crypto works it requires that all 3 parties communicate. There are public and private keys, some that are and are not transmitted. Basically the way it works is through some magic mathematical crypto functions where a private key and a public key can decrypt the transmission. Some passwords are public, but some aren't. It's a little bit difficult to explain, but it basically hinges on the fact that you are communicating with a third trusted party, and the transmissions are enctypted.

There is such thing as an SSL strip hack, but the users or the web page should be coded in such a way that the web page fails to display something, or the user should notice that their browser is not communicating with a crypted site.

Yes, you should not assume that the user is a good guy either, I was just explaining that forensics should be able to tell if someone is definitively a bad guy. A few times my home net has been through the grinder, and articles of law that say "the user is responsible for the security of their own network" sometimes stick in my craw, especially where some of the techniques used would have been virtually impossible to defend against anyway. sorry. You're right though security should assume that everybody is the bad guy first, before they authenticate, law should assume that everybody is innocent until proven guilty. That's what I was getting at. Sorry for the bad tradecraft.

Something you might be able to use in a corporate environment is an intrusion prevention system, but the net admin would have to set it up. Also you could set up what are called application servers, which from what I can tell (I'm still a little bit new to corporate level code) are servers that basically screen the BS from transmissions before forwarding to the DB. I am guessing if you really really want to you could code your own IPS, with a sniffer component which parses the packets to ensure validity before passing them on. As to what you should parse for, I don't know entirely. Apparently it should take into account which tables the user/web page actuallly has a need to access, and it should screen out anything that looks suspicious, ex large amounts of NOP characters (no operation). That is called an NOP sled, which is used in exploits. This might not be what you're asking exactly. Other things you could look into are SQL injection attacks (you probably already know about them), but basically how they work is the programmer foregets to screen them out, or format the injected string to act as a string when it hits the database. I think it is basically formatting the quotes correctly, or formatting the string to another format. I'd have to look the solution to that one up, I did actually know at one point. Looking up the attacks typically spreads a lot of light on how to avoid them, just try to browse via reputable sites, or use a VM/sandbox while browsing.
-Buffer Overflow (C) - there are different types of this...
-Format String Exploit (C)
-Sql Injection (DB-related)

Screening out characters that typically wouldn't be unlegible would also be a good idea, since they would then be exe instructions. For instance non-legibles in a data portion which would normally be encoded as legible text.

Typically the key to security is collaboration, and layering. More layers limit more of the attack surface. I hope this post strikes closer to the mark.

[Aeonix]

I still don't understand.

Couldn't owner of the computer just take the private/public key, decrypt the message, analyze it, then create program which would send same queries (to the server)?

That's what I'm trying to prevent, and apparently there are such companies, that can do it. Just I don't know how.

If I write
{"message"=>"Hi!"}
and they decrypt it and replace with
{"message"=>"Here's a virus for ya http://googol.ex.fr"}

I want to prevent people from using automated programs to execute tasks (commands) on the server. Of course I could analyze packets of data, but it's not like programs can't mimic it.
{"message"=>"Here's a virus for ya http://googol.ex.fr"}
How do I verify on that, that program hasn't sent it, but user? Comapnies do it, I need to know how >:C

It seems you're concentrating on safety of the content. Once again, I'm assuming that content is HEALTHY. First, I need to detect that sent request is legitimate and not automated (this is the point of the topic), then I can bother with protection.

[rproffitt]

@A,

Stick with HTTPS. To understand why man in the middle attacks are hard we would have to hold a class on it all. There are long priors such as https://www.google.com/#q=https+man+in+the+middle+attack https://en.wikipedia.org/wiki/Man-in-the-middle_attack and then we have SuperFish.

Now you could create your own checks but for most, https will do.

[Aeonix]

Stick with HTTPS.

But...

Couldn't owner of the computer just take the private/public key, decrypt the message, analyze it, then create program which would send same queries (to the server)?
.

How would server verify that the content hasn't been modified? If I have the content and en/decryption key, I could forge new hash, couldn't I?

If I stick with HTTPS, couldn't... can't I write? Couldn't owner of the computer, manually decrypt the message, find out how it looks, and recreate it and then automate queries by program?

I know that HTTPS protects me from middle man attacks (which I do understand), but I'm worried about computer owner himself/herself. I don't want one guy to create program that would send queries back and forth with my server. Sending everybody copy of "Hi, visit this link (link)" to 128 people in contacts list, with just single button.

[rubberman]

Your private keys are typically kept in an encrypted file that uses a key that only YOU know. Yes, if someone has placed a trojan in your system, they can get all this stuff. If that has happened, then you are well and truly pwned and all your secrets are their's. If not, then you are safe and your communications are secure.

[Aeonix]

then you are well and truly pwned and all your secrets are their's. If not, then you are safe and your communications are secure.
So. I'm pwned either way. The user does have the key.

[rproffitt]

@A,
You lead with "Verify if data sent by servers?" and have changed to where the user is sending the data. There is no accepted ironclad secure comms today. You may askself why this is and the answer is out there for you to read. Start by reading the Wikipedia on Lavabit.

Governments do not want unbreakable cypto. If you want to create such a thing, go right ahead but use what happened to Lavabit to foresee what may happen next.

[ryantroop]

Disclaimer: This is my understanding of how all this works - if I am mistaken, or uninformed, I would love to be better educated on the topic.

So the way you as a programmer defensively do this is to use a method similar to oAUTH 2.

You have a package (your data), which is encrypted with a key. The bundled package is sent along with the unencrypted state (or you can encode it or whatever you like (base64 is popular)), as well as the PUBLIC key for the recipient machine to decode your data, and the encryption method (which you can ommit if you know the method since it's your application, but obscurity will only get you so far).

Over HTTPS, this will go a step further and do a handshake to verify the machine that it is receiving data from is consistent. As far as Man in the Middle attacks, you will forever be vulnerable because you do not set up direct connections between machines when communicating over the internet. There are routers, load balancers, and every other type of machine out there. It will happen. It does happen. Move on with your day and code defensively to expect it to happen.

Now, lets look at what your little data package is gonna do. Since it is encrypted and encoded, the man in the middle would have to be able to do a number of things quickly. Intercepting the data is one thing - since you have to send the unencrypted data over the wire, it's freely accessable. So read access happens unless you also have a method for encryping/decrypting this portion (It would seem the internet at large feels that HTTPS/(SSL or TLS) covers this pretty well).

For guaranteed arrival, though, you now have the package in an "clear" unchanged state, as well as your packaged (encrypted) data. If you unpack the encrypted one and it matches the other, you are pretty guaranteed that there has been no interruption along the chain. The reasoning for this is that encryption and decryption are meant to be slow, and many of them have independent time variables which are used (like timestamps). The man in the middle attack would theoretically take too long to intercept, gather, rewrite, repackage, and then transmit the modified data. On top of that, if it is over SSL/TLS, they would also have to decrypt the encoded string being sent in the first place (which, in theory, only governments have the resources to do right now - so if they can do this, the world has bigger problems than your theoretical app).

So that's that... If I missed anything, or misrepresented, please correct me.

Toodles!

Ryan

[Aeonix]

I can't stop laughing.

Why does everybody focus on "Middleman Attack" and protection of sent data? That's not what I'm asking xD
Did I really mistyped the post and title and everything else, so badly, nobody seems to find out what I'm meaning to ask?

Okay, forget everything you have as assumption about my problem.

I have a server, which receives requests and sents responds. It's type of a program. It will get order, for example [send="Aeonix",content="Hi!"], the server will then send content of this message to, well, my program, and program will notify me about it, as programmed. Clicks within the program links (like buying premium or something), uploading configuration files (some preferences online). Are all send back and forth. It's all great and all good.

Now, there's that rearsniffer, he's a programmer, he takes the private key from his computer, manually decrypts server's message. Learns how queries are built (their logic). He finds out, that he could manually resend the query, change the date, change the content and tries [name="RogueTestingAccount1",content="Hey, visit this virus.com!"]. And suddenly it works! He decides to exploit the system, with recommended friends, instead of actually clicking for links, he creates a program, logs in, receives the list from server and instead of [addFriend="NameOfSuggestedFriend"], he uses program to add them all automatically by parsing the list.

I want to prevent such automated use. How can this be done?
How do I know? Example that I used in first post:

I was playing an online video game, it's quite popular and out there. On some day, a programmer released his very own version of the launcher (with buttons, shop, and ways to enter a game (which still would open the original "battle arena .exe")). Owners of game didn't like it. And even though program sent exactly same query as the original launcher, still, majority of users of this "new launcher" got banned for 3rd party software (as it was forbidden in ToU).

How did they prevent it? It's not like just adding token or changing it, would prevent that programmer from creating such app.

I know that no method is 100% secure. But hashing passwords is better than leaving them plain. So there must be a better method of protecting this, than leaving it as it is. I really don't like the idea of automated usage of my server. Especially when it's meant to be used by human, not by robot.

I could parse it so that if program adds 100 friends in 1 second, that means they're botting. But they could also set interval.

Off-topic, doesn't contribute to the understand of topic.

Start by reading the Wikipedia on Lavabit.
Would that matter if I'm from EU? I mean, they were Texas programmers offering US citiziens uncrackable webmail. It's all within US, does NSA even have official legal power here? I mean. I know that local army (here in EU) can force me to show one's webmail. But they can't really snoop on my connections 24/7, right? And what about countries where there's no detention laws (they still can request to see mailbox, but can't install surv. software), like Japan, Romania, Serbia, Ukraine?

[ryantroop]

Likely because his personal private key did not match the key on the server, and the server did not originally care to authenticate. With a patch, they suddenly care, and then were able to block users due to malformed or incorrectly encrypted data.

Or, the encryption type used was not very good (such as MD5), and was easily cracked by modern standards. If they updated their encryption method, they could simply look at the resulting login string to know which hashing algorithm was used and ban/deny any that came from MD5... without knowing their code, and their specific setup, it's impossible to know what method of intrusion detection they have, or how they were able to figure out where a bad request was coming from.

[rproffitt]

Yes, your topic is "Verify if data sent by servers? How do bigger companies do that?" so we get to dive into a lot of the MITM.

Later you clarify the messages are not coming from servers and we are now in the weeds as you now have a new topic.

I like what ryantroop wrote but you need to be clearer next time. You lead with data sent by servers and changed that to something else.

My note about Lavabit is to start to learn why we don't have airtight crypto on everything. Even with your reply it's not off topic at all. It's a big topic until you re-state your topic and question. Writing a secure comm is one thing, and here I think it best you try again with a more refined topic and question. As it stands your data reply is not from a server. Try again. I will.

[ryantroop]

Ok, simple answer for you - they use TCP. By design and specification, TCP is required to reach the target in the specified order it was sent. In contrast, UDP does not, and therefore is not "guaranteed." For added security, they "sign" their data (much like a real-time checksum).

Does that answer your question?

[overwraith]

Since there are public and private keys on all three computers, of which private keys don't get transmitted, and we are talking about encryption which by it's nature protects data in some state or another from tampering, you can be fairly sure that the destination has recieved it's contents unfettered. The thing about encryption is that it essentially transmits a stream whereby if any single byte is removed, the entire transmission becomes corrupted, and is dropped, and then retransmitted. It's like playing music in the background when you presume that you are being audio recorded, so that people can't copy and paiste bytes where they choose them to be. The background music changes would be noticable. Now the key store question, sure somebody could probably make some sort of script which could access this, somebody else would know more about whether that was possible or not, but the question here is not how to protect it on the destination computer, it's how to potect it in transit.

[Aeonix]

But... everybody is still talking about middleman attacks :D

Yes, your topic is "Verify if data sent by servers? How do bigger companies do that?" so we get to dive into a lot of the MITM. Later you clarify the messages are not coming from servers and we are now in the weeds as you now have a new topic. I like what ryantroop wrote but you need to be clearer next time. You lead with data sent by servers and changed that to something else.

Yes, I know, my bad.

Likely because his personal private key did not match the key on the server, and the server did not originally care to authenticate. With a patch, they suddenly care, and then were able to block users due to malformed or incorrectly encrypted data.

Can't the rearsniffer obtain that private key from his operating system? And change accordingly to DO match the changed private key? Doesn't that "de-resolve" it?

Since there are public and private keys on all three computers

But, there are 2 computers. Server and user.

and we are talking about encryption which by it's nature protects data in some state or another from tampering

Yes. But owning local private key, and public key, doesn't that allow de/encrypt the message? And reforge the hash for verification?

The user has both private and public key on their computer the moment we did Handshake. They can easily manipulate the data.

I give up. I messed up a question, now everybody's talking about Middleman Attacks.
xD

[rproffitt]

Just start a new topic. Be clear what the question is or we'll repeat this discussion.

In parting, anyone can lie to you today. I wish your topic and question was more about what you really wanted to ask. Then we could talk about that.

[ryantroop]

The whole point of the private key is that it is stored only on the desitination server (or, in oAUTH's case, it is also stored on the Provider's box who acts as the authenticator, so it's in 2 places) - i.e., it is private. If the user in question got a hold of the private key, you likely have other security breeches to worry about (like, the fact that they have user access to your destination machine, or the machine of the Provider).

It sounds like you are fishing for flaws in security systems. There is a reason we rely on standards, and what everyone is telling you is current best practices and standards - that's because they are, for the most part, vetted and proven safe. Regardless if we are talking about MITM attacks or not, the method of securing the transmitted data remains the same. Somehow, some way, you encrypt the data you want to send, and it is unencrypted on the other end. If it doesn't match up or is somehow broken, it's considered insecure. End of story.

In regards to your "hypothetical," it is impossible to know how the individual in question did what they did without some form of forensics to back up the claim, or some sort of proofing that duplicates his actions. It's really just that simple - there is no way, short of a mental exercise in futility, that anyone here will stumble upon the exact way someone cracked one single stream and figured out how to replicate. We can make educated guesses, but that's about it.