While I like projects and such work, I continue to run into folk that think these are a panacea to what ails product development. You can make bet on my view on that.
As to the game engine, since you will be at it for years, it's not a bad way to go. IOW, go for it. My parting advice is to the would be game maker is to never create a new engine. Re-evaluate existing engines.
I agree, it's a variant of agile. I've used agile-type methodologies in projects since the mid 1990's, and I'm a huge fan.
The truth of software development is that nobody really knows what the product is supposed to be until they see it and use it. That's why waterfall always fails.
Keep prototyping and getting feedback on the prototypes until the feed back is good enough. Then tidy it up and you have version 1.0 ... at which point you haven't finished, you just started.
OK, I asked a general question and got very general answers. @rproffitt's advice on not reinventing the wheel is quite sensible - but I want to anyway. (Because I can and it's fun!)
Maybe I should elaborate. I'm looking to write a 2D platformer in C++/SFML. I know I'll need to simulate key-presses and mouse-clicks and test if the desired outcome is achieved: did the character move? did he/she not move? I guess was hoping for a little more feedback on people with relatable stories.