I've been studying games at uni for the past 2 years, and if you want to do any graphics programming (especially 3D graphics), I would highly recommend learning some of the maths first. I'd focus on geometry, vectors, matrices, and their application to 3D graphics/physics. This will make learning the graphics APIs and programming a lot more straight forward.
If you want to jump in the deep end and want a really good reference book, I highly recommend "Introduction to 3D Game Programming with DirectX 9.0c - A Shader Approach" by Frank D Luna. It was our prescribed text for learning shader programming, but it has a little bit on maths and programming that will at least point you in the right direction if it's not enough. Obviously it's DirectX, so Windows only.
Having said all this, everyone (including me so far =P) seems to be assuming you want to program fairly "serious" 3D games, which generally means C/C++ with OpenGL or DirectX. If you're learning yourself, you might find it more rewarding and easier to get started with a higher level programming language like C#, Java or Python and program some basic 2D games like tetris or space invaders. As pointed out above, a framework like XNA could be a good place to start. Once you're more comfortable with programming, and basic games concepts, then you can move onto C/C++ and graphics APIs if you feel like it. Learning 3D maths, graphics, and C/C++ all at once could be quite daunting, so starting out small is probably a good idea.