There is probably a way to make it work as a function within the function. However, this is really unusual and unnecessary. The proper solution, seconding NathanOliver on this, is to make the recursive function a private member function of your class.
Also, having those static local variables like that is a bit weird, too weird for my taste. In fact, it's a bug because if you call the same function more than once, then those static local variables will not be initialized again (will keep the values they had after all the recursions of the first call. Personally, I think you should just create a private nested class within your class to do the recursion. As so:
class RMatrix
{
int sizeH;
int sizeL;
int TotalN, Replace;
class RepMatrixCalculator {
private:
RMatrix* parent;
int Row;
int* RepIndex;
public:
RepMatrixCalculator(RMatrix* aParent, int MAX) :
parent(aParent),
Row(0),
RepIndex(new int[MAX]) { };
~RepMatrixCalculator() { delete[] RepIndex; };
void RepMatrix(int NDR, int size, int PosIR, int calls);
};
protected:
bool ** Matrix;
RMatrix(int, int);
~RMatrix(){for (int x = 0; x<sizeH; x++)delete [] Matrix[x]; delete [] Matrix;}
void DisplayMx ();
void RepMatrixf();
};
void RMatrix::RepMatrixCalculator::RepMatrix(int NDR, int size, int PosIR, int calls)
{
if (calls==0)
;
//memset(parent->Matrix, 0, sizeof(parent->Matrix));
// NOTE: The above line of code is completely wrong.
// NOTE2: Setting the matrix to zero probably something you could do in
// the constructor of RepMatrixCalculator.
for ( ; PosIR < size-(NDR-1); PosIR++)
{
RepIndex[calls]=PosIR;
if (NDR==1)
{
for (int x = …