That is a problem, yes. That's why public keys are cryptographically signed (look this term up) by third parties. For example, the 'certificates' that websites using https have are signed by Verisign and other organizations that browsers know about.
Suppose you're friends with Ken and Ken's friends with John. Since your Ken's friend, you and he have traded public keys, and since John's friends with Ken, he's traded public keys with Ken too. When John traded his keys with Ken, well, Ken took the opportunity to sign John's key, saying that, yes, this is in fact John. So then when John sends you his public key (along with Ken's signature), you can see that, hey, it truly is John, and you know so because Ken said so. And he said so in a cryptographically secure fashion. Now hopefully, Ken's signed your key, and that way John can trust that you are who you say you are. Sometimes (usually) that's not necessary -- maybe John doesn't care or maybe John has other ways of recognizing you, like, by asking for a password.
This is what nerds with no life do at key signing parties.