Python C++ extension (multi threaded environment)

[alokjadhav]

I am using SWIG to extend Python to C++. Understand that there is a different forum for SWIG users but I am still waiting for a response on that list. But the questions I have consists of basic Python C API releated. I hope this list will be able to help me.

I am trying to solve a simple problem which many of you must have encountered. I wonder what I am doing different than others that I am facing a problem.

Background
-----------

A c++ library is wrapped in my C++ Class. This new C++ class is extended in Python using SWIG. I have
to call a python function from C++ code ( which is on a new thread spawned from C++ main thread).

Issue
-----
I am able to call the function. No issues. I could pass basic datatypes such as int. No issues. But when I try to pass a C++ class object (eg RFAMessageWrapper) to my python function I get an exception. It is an Access violation exception. The memory could not be "read". I am not even able to catch this exception.

Some code snippets
--------------------

C++ function which calls python function. ...

static void PythonCallBack(RFAMessage *msg, void *clientdata)
{
   PyObject *func, *arglist;
   int blah = 1;
  
   PyGILState_STATE state; 
   state = PyGILState_Ensure();
   func = (PyObject *) clientdata;               // Get Python function
   RFAMessageWrapper msg2(msg);
	try {
		arglist = Py_BuildValue("(O)",msg2);             // Build argument list
		//arglist = Py_BuildValue("(i)", blah);
		PyEval_CallObject(func,arglist);     // Call Python
	} catch (...) {
		cout<<"Unknown exception..."<<endl;
	}
  Py_XDECREF(arglist);                           // Trash arglist
   PyGILState_Release(state); 
}

Python function call..

def callback_fun(msg):
    try:
        print "RFAMessage received for service:"+msg.service
    except Exception:
        print "Exception handled"

Whenever I try to access msg (of type RFAMessageWrapper .. C++ object) I get exception. msg is not None. I checked that in if condition. even type(msg) raises exception.
RFAMessageWrapper class has been wrapped properly by SWIG as I could create an object manually and use it well in Python. But only when I pass it from C++ to python I get this error.

I have spent long time to solve this issue but in vain. I hope I get some help from this Forum.

Regards,
Alok

[Gribouillis]

Im not sure, but I think you should wrap msg2 into a new python object yourself with SWIG_NewPointerObj . You can put only python objects in a call to a python function. Try to google for use examples of SWIG_NewPointerObj. You must pass your c++ pointer and a swig reference to the wrapped type.

[alokjadhav]

Im not sure, but I think you should wrap msg2 into a new python object yourself with SWIG_NewPointerObj. You can put only python objects in a call to a python function. Try to google for use examples of SWIG_NewPointerObj. You must pass your c++ pointer and a swig reference to the wrapped type.

I used SWIG_NewPointerObj method to create a PyObject. It removed all my exceptions and seems to be working fine.

PyObject *argobj = SWIG_NewPointerObj(SWIG_as_voidptr(&msg2), SWIGTYPE_p_RFAMessageWrapper, SWIG_POINTER_NEW |  0 );
    try {
        arglist = Py_BuildValue("(O)",argobj);             // Build argument list
        PyEval_CallObject(func,arglist);     // Call Python
    } catch (...) {
        cout<<"Unknown exception..."<<endl;
    }

Callback is called.
But now, in Python code

msg.service raises exception.

Python call back is

def callback_fun(msg):
    try:
        print "RFAMessage received for service:"+msg.service
    except Exception:
        print "Exception handled"
        dir(msg)
        print type(msg)
        traceback.print_exc(file=sys.stdout)

Result

Exception handled
<type 'SwigPyObject'>
Traceback (most recent call last):
  File "<stdin>", line 3, in callback_fun
AttributeError: 'SwigPyObject' object has no attribute 'service'

I am expecting the object type as RFAMessageWrapper here. Even when the PyObject created in C code uses SWIGTYPE_p_RFAMessageWrapper as the typeinfo, still the object created is not RFAMessageWrapper.

[Gribouillis]

I don't know the solution, but I once designed a hack which you should try: You declare a static variable somewhere and an accessor function without arguments for this variable like this

RFAMessage *theVariable;

RFAMessage* getTheVariable(){
    return theVariable;
}

Now you declare getTheVariable in your swig interface file, and swig will generate a python function getTheVariable() which returns a python object wrapping the pointer read in theVariable.

Back to your c++ code, you store a pointer to this python function generated by swig and you can use it to wrap a pointer

PyObject* getTheVariablePythonVersion;

PyObject* mywrapRFAMessage(RFAMessage* msg){
    theVariable = msg;
    return PyObject_CallFunctionObjArgs(
                   getTheVariablePythonVersion, 0);
}

There may be thread issues if different threads want to use theVariable. It looks complicated, but it should allow you to wrap an object without using swig's api.

Also you can look at the code generated by swig for the function getTheVariable and this should tell you how to properly wrap a RFAMessage pointer.

[alokjadhav]

Thanks for your response. I understand my mistake. But I am still not able to solve the issue.

Above code will not work in my case because your design is suitable for singleton pattern. Where as I don't have single object of RFAMEssageWrapper. 1 object is created for every new message i receive.

In my C++ callback function below

static void PythonCallBack(RFAMessage *msg, void *clientdata)
{
   PyObject *func, *arglist;
   PyGILState_STATE state; 
   state = PyGILState_Ensure();
   func = (PyObject *) clientdata;               // Get Python function
   RFAMessageWrapper msg2(msg);
 
   PyObject *argobj = SWIG_NewPointerObj(SWIG_as_voidptr(&msg2), SWIGTYPE_p_RFAMessageWrapper , SWIG_POINTER_NEW |  0 );
  
   arglist = Py_BuildValue("(O)",argobj);             // Build argument list
   PyEval_CallObject(func,arglist);     // Call Python
  Py_XDECREF(arglist);                           // Trash arglist
   //SWIG_PYTHON_THREAD_END_BLOCK;
   PyGILState_Release(state); 
}

Can you tell me if the usage of SWIG_NewPointerObj is correct? I want to convert a C++ class object into python equivalent object. In the python code, the objetc is identified as SwigPyObject and not RFAMessageWrapper which I am expecting.

How can I convert RFAMessageWrapper object into its Python equivalent to that in my Python function I can use the passed object as RFAMessageWrapper.

Regards,
Alok

[Gribouillis]

You didn't understand my design above. It's not a singleton, the variable is only used temporarily when wrapping an object (by the way, I should have used RFAMessageWrapper instead of RFAMessage).

I can't tell you if your usage of SWIG_NewPointerObj is correct, but you can ask SWIG
how it wraps a RFAMessageWrapper*. For this as I said above, you can declare a pair

RFAMessageWrapper* foo;

RFAMessageWrapper* get_foo(){
    return foo;
}

(just for testing purposes) and you declare the function

RFAMessageWrapper* get_foo();

in your swig interface file. Then your run swig and you look in the c++ program generated by swig how swig wrote the wrapper function wrap_get_foo (or a similar name). The code for this function shows you how swig makes a python object with a RFAMessageWrapper*.
If you follow my instructions, please post the c++ code of the wrapper function.

[alokjadhav]

The generated code for the wrapper is

extern "C" {
#endif
SWIGINTERN PyObject *_wrap_GetRFAMessageWrapperObject(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
  PyObject *resultobj = 0;
  RFAMessageWrapper *result = 0 ;
  
  SWIG_PYTHON_THREAD_BEGIN_BLOCK;
  if (!PyArg_ParseTuple(args,(char *)":GetRFAMessageWrapperObject")) SWIG_fail;
  {
    SWIG_PYTHON_THREAD_BEGIN_ALLOW;
    result = (RFAMessageWrapper *)GetRFAMessageWrapperObject();
    SWIG_PYTHON_THREAD_END_ALLOW;
  }
  resultobj = SWIG_NewPointerObj(SWIG_as_voidptr(result), SWIGTYPE_p_RFAMessageWrapper, 0 |  0 );
  SWIG_PYTHON_THREAD_END_BLOCK;
  return resultobj;
fail:
  SWIG_PYTHON_THREAD_END_BLOCK;
  return NULL;
}

If you see the function The line

resultobj = SWIG_NewPointerObj(SWIG_as_voidptr(result), SWIGTYPE_p_RFAMessageWrapper, 0 |  0 );
  SWIG_PYTHON_THREAD_END_BLOCK;

is precisely what I have used. It returns a PyObject *, but it is not the one I am looking for.

The returned PyObject * points to a SwigPyObject type.. which is a C++ pointer to My RFAMessageWrapper object. I cannot use this object directly in my Python code. I have to use a Proxy object (already defined in Python using swig) to wrap this pointer.

I can create a new object of Proxy class which creates a new pointer to C++ object but I don't know how can I create an object of a proxy class if I already have a pointer to C++ object. (the value returned by SWIG_NewPointerObj in my case. )

So I am still stuck here :(

[Gribouillis]

Perhaps your problem comes from the fact that you are trying to access an attribute. You should try to define an accessor method

class RFAMessageWrapper {
    ...
   ??? getService(){
      return service;
  }
  ...
};

and call msg.getService() ?

[alokjadhav]

I SOLVED IT ...
thanks for people who provided me help .. but i believe solved it myself. .haha .. took me long time..

so .. the solution is something like this..

in my c function ..which calls python call ..

PyObject *argobj = SWIG_NewPointerObj(SWIG_as_voidptr(&msg2), SWIGTYPE_p_RFAMessageWrapper, SWIG_POINTER_NEW |  0 );
 try {
  arglist = Py_BuildValue("(O)",argobj);             // Build argument list
  PyEval_CallObject(func,arglist);     // Call Python
 } catch (...) {
  cout<<"Unknown exception..."<<endl;
 }

I am using
PyObject *argobj = SWIG_NewPointerObj(SWIG_as_voidptr(&msg2), SWIGTYPE_p_RFAMessageWrapper, SWIG_POINTER_NEW | 0 );

This was incorrect... i guess SWIG_POINTER_NEW is used to create pointer to the actual c++ object where as if I pass 0

PyObject *argobj = SWIG_NewPointerObj(SWIG_as_voidptr(&msg2), SWIGTYPE_p_RFAMessageWrapper, 0 | 0 );

This actually creates a pointer to the wrapped python object :)

I could access my object in my python code now.

I had to include std_string.i to enable string typemaps :)

Thanks,
Alok

[Gribouillis]

Congratulations ! Can you mark the thread as solved ?

[alokjadhav]

The problem i was trying to solve was to call a python function from my C++ code. I was using function pointers to achieve this. Someone suggested that I should use directors in SWIG in which i can write virtual classes in C and then implement those classes in Python. That would be more OO way. I am going to work on this next.

Thanks Gribouillis for your time.