double *solve( double a[ ][N], double b[ ])
what does the pointer notation * represent in front of solve?
It means the function is going to return a pointer to a double variable or possibly a pointer to an array of doubles. The * means a pointer.
Read it as if the * sticks to the type name, not the function name.
Pointer syntax in C/C++ is very flexible, but likewise very confusing. You can say something like
int i, *pi, **ppi; but this should be read carefully as
int i; int* pi; int** ppi;
:an integer, a pointer to an integer, and a pointer to a pointer to an integer. Yeah, I know... read it again anyway... (I attached the "*" to the int because it is part of the type of each variable, but it could be anywhere between the base type and the variable name.)
In C, you are forgiven a lot when dealing with the type of things. Hence, as already mentioned, the variable
int* pi can be a pointer to a single integer or a pointer to a whole lot of them. Typically, programmers will state their intent by saying either
int *a --pointer to a single int, or
int a --pointer to an array of ints.
However, you can't use the  brackets when declaring a function return type, so you are stuck with the *.
Hope this helps.