Java is easier to debug than C++, unless you already know C++ inside out, C++ is a tiny bit faster at matrix calculations, otherwise java and C++ are comparable, they both use DirectX and OpenGL interfaces, and the VirtualMachine layer is passed-through very quickly. I'd rather have portable code that takes 10 extra nanoseconds per minute than native code that doesnt. (my time relationship there is based on nothing but unfounded conjecture)
VB is actually not bad if you use DirectX-in-VB, I made some 2D a long time ago, and there's certainly no loss in potential from using what some people would claim is a "restricted" language. I've never tried DirectX3D in VB, but I think it's supported.
EDIT: to be honest, you'll be learning DirectX or OpenGL rather than a programming language for the graphical part, and they're both similar. The programming language you use is more the programming language you think is best for implementing the object relationships and general aspects of the game engine. Pick your poision based on the packaging.
Matt