The sum of the sqaures of the 2 larger interagers is 12 less than 4 times the square of the smaller number.

This is definitely not a Java problem but a word problem. Though, I will help you out a bit this time. Break the question down into parts.

1)sum of squares of the 2 larger integers
2)4 times the square of the smaller number
3)12 less

So there should be 2 integers (unknown variables) in this equation. The 12 less means that the total value from #1 is 12 less than the totale value of #2; therefore, you could rewrite #2 as
2)(4 times the square of the smaller number) + 12

OK, you should have gotten what you want now. If you cannot do word problems, you may need to retake whatever math class you are taking...

2 integers? Sounds like 3 to me (2 "larger", 1 "smaller") So is this like
6^2 + 4^2 = 52 = 4x 4^2 -12
which fits the math but doen't have a unique "smaller" number?

Eh? What are you talking about James? :) You got the equation correctly, but why you are talking about another number? The question is asking for 2 integers that will balance the equations given by the problem. You already gave one correct set of answer (4 and 6). Therefore, why do you need another integer for? :)

PS: There are more than 1 set of integer to answer the problem. The problem is asking for an equation which could be used to find an answer. I am not going to write out the equation, but some answers are (4 & 6), (14 & 24), (2704, 8100), etc.

It's just an ambiguity in the text - why refer to the "two largest numbers" if there are only two numbers in the equation? Surely you would only say that if there were >2 numbers?
Anyway, I was just chipping in in case it helped.

OK :) Well, I will give the equation for the problem then...

/*
S <- smaller integer
L <- larger integer
(S^2) + (L^2) = (4*(S^2)) - 12
L^2 = (3*(S^2)) - 12
or
L^2 + 12 = 3*(S^2)
*/

Hope this help you to see that there is no third integer because it is being omitted by the equation and it is useless in the sense of finding an answer set.

How can I make this to continue until the user presses N? I've tried using while(1), while( option == 'y' || option == 'Y') but it kept giving an error ...