The Function

double *solve( double a[ ][N], double b[ ])

what does the pointer notation * represent in front of solve?

It means the function is going to return a pointer to a double variable or possibly a pointer to an array of doubles. The * means a pointer.

Read it as if the * sticks to the type name, not the function name.

Pointer syntax in C/C++ is very flexible, but likewise very confusing. You can say something like int i, *pi, **ppi; but this should be read carefully as

int i;
int* pi;
int** ppi;

:an integer, a pointer to an integer, and a pointer to a pointer to an integer. Yeah, I know... read it again anyway... (I attached the "*" to the int because it is part of the type of each variable, but it could be anywhere between the base type and the variable name.)

In C, you are forgiven a lot when dealing with the type of things. Hence, as already mentioned, the variable int* pi can be a pointer to a single integer or a pointer to a whole lot of them. Typically, programmers will state their intent by saying either int *a --pointer to a single int, or int a[] --pointer to an array of ints.
However, you can't use the [] brackets when declaring a function return type, so you are stuck with the *.

Hope this helps.

commented: pretty decent breakdown +1