[Python]Create dynamically loadable APi

[tzushky]

Hello,

I've been swimming with Python for a while, but there are remote field of it which I don't find easy. For example I have made an API from some XML files, and there will be many versions of it. The API is actually several modules either with functions or simple value definitions in python. So, when I use it, I'd like to e able to dynamically load whichever version I want just by entering its name. I've looked into dynamic module loading, but it's not exactly what I need.

If it's not very clear, I'll try and enhance,

Thanks,

T

[slate]

Yes, please, try to elaborate on the problem.
Dynamic module loading can mean different things, but I think you need that.

For example wxpython uses the semantics:

import wxversion
wxversion.select("2.8")
import wx

You can use the following to import a module given on the command line:

exec "import "+ sys.argv[1]

[tzushky]

Thanks for the version , I didn't stumble across it yet...
I'll elaborate...

The fact that it is an API is less imporant probably....I found a silly yet easy way to get to the files I need, but the import is the problem due to scope. I'll explain.

Say the API is in D:\dev\v1\myAPI, (or D:\dev\v2\myAPI...etc for several versions)

Say I enter v1, or v2 at execution so I know were to look for the files (I really want to import everything in the myAPI folder with from...myAPI import * because I need the functions and variables defined there for direct usage (not module.nameFunc...it's too long)

I can use the glob or dircache modules to get the list of files and then create the exec import_string for each. The problem is where I do this import.

Because my file with the import function (I built one because it looked neater) if elsewhere, say in D:\dev\myExec

And there I also create an instance of a class which inherits a class in the API, but need those imports I make... Only the imports stay in the locals() dictionary in the import function in the executable, and do not stay available for all from now on....I think I have to find another way, or put them in the globals() but I don't know how...

Recap:

- D:\dev\vX\myAPI contains one module with ClassToInherit
- d:\dev\myExec contains the main, which calls a function F1, which calls the import function and then makes an instance inheriting class to inherit.
Only the imported stuff stays in F1's locals()...and I need them both in myExec and for the instance to exist (the ClassToInherit is not found in the symbol table outside F1 and not even in the globals() of F1)

Any better?

[tzushky]

I've kept reading and reading... I'm trying to do something very strange once from what I understand: because my imports put my new names into the namespace of a function, and I would like these names to be known at the same time:
- inside the module I import next (which is probably impossible since a module has it's own namespace and I haven't found a way to make it aware of an outside namespace)...

- and also, the names whould have to be available in the main namespace too....

Such special situations I haven't encountered yet, so...unless a solution is found, I'll change the whole thing...

[tzushky]

I kind of posting to myself here, but I think I've found a way around half the problem at least:

I read that when you import a module, the object with that name is put into the sys.modules dictionary.

So before anything, in the module containing the class I instanciate I check sys.modules for the name of the module containing the ClasstoInherit .God, this is hard to explain.

myAPi--
|----api_fileN
|--------ClassToInherit (many
functions no init)
|---------also calls a function defined
in its daughter (circular, I
know)

myExecFolder
|-------DaugtherClass(ClassToInherit)
|-------myExec
|---Main
|-----calls Func1
|-----calls imports (no separate import function anymore to avoid losing to locals()
|----then makes Daughter
instance and needs to
call functions from
ClassTOInherit

So, at DaugtherClass file beginning, before its declaration, I do the test for the sys.modules content and exec 'from TheModuleIWanted import ClassToInherit'

problem is, that with a manual import, using the DaughterClass function in the Parent worked, now with the exec import, I doesn't recognize the Daughter function in the Parent class...how do I avoid complete execution?...(because I think module execution for completing the Daughter namespace is the problem)...

This is the next step...

Phiuh...going to lunch now

[tzushky]

How about defining empty functions with the same name in Parent Classtoinherit?.. preliminaries show no more errors, now I have to test the whole thing....

[tzushky]

Still me for me...if anyone else is still interested though here it goes...

The sys.modules dict proved useful, I use it to re-import (although not reload() sense ) the modules I find in my executable main, into the Daugther class before its definition...And I don't seem to need those functions after all, I don't know why at run time It had given me that error...

So, so far so good... I'd like to make this thread solved if nobody else minds or finds another tweak to this problem...

[slate]

I am interested.
Can you give me a stripped down example? If you have time for that.

I have read the problem description several times, but could not understand it.

Maybe your solution is helping me to understand it, and recommend another hopefully better one.

[tzushky]

I'm not sure what stripped down is, but I'll give my example. just grab a pen, I'll probably make too many phrases and u might get lost in all of it.

I obtain my API files from C header parsing, so I separated into
- Defines.py,
- Enums.py and
- Structs.py

Then there are some XML files which help me generate some functions (they are for sending packets). The decision was to make a class containing these functions: the ClassToInherit (ApiFunctions). It's defined in a module called Macros.py

So you have your folder=package called pyapi with
- Defines.py,
- Enums.py,
- Structs.py and
- Macros. py containing the magic class...

I also have a class Device which needs to inherit the class in Macros.py and also use Defines and Enums and Structs (structs contains functions too by the way, long story there...)(this is where the dynamic imports should arrive, at the top of this class' definition)

Then I build a client in python for a server I open. The client is actually a larger application, the connection is the smallest part. In the client I make the socket, connect it, give it as parameter to the Device instance constructor. But before I make the instance I must import because this is the level where my API version number is available.

But when I import the pyapi package modules(as all Python reference well explains) the imported names stay in the function /module namespace, I couldn't possibly count on them to be known inside the module containing my Device class.

So I used a trick: sys.modules keeps all names of defined modules at runtime. even if they were or not well imported in the client, I have the names in sys.modules, so I make the imports using those names at the top of the Device.py file.... Now the ClassToInherit name is known, so the Device instance is made and all functions and constants from the aPI are seen in both namespaces: client's and Device's instance...

This still seems horrible...I really took the time to read all the references of exec 'from XXX import *' or using __import__, etc... I really understood where imported names are created and how they stay in local namespace or global of the module where they are imported.

but I really don't see another workaround the fact that I import module A and B in C, and would like to make A available in B at its import...this would be the schematics...

Any better now?

[slate]

I think this is the case where metaclass is usable.
I've never thought that it is usable for anything at all:)
I am happy that I see a real use case for it.

I look up the metaclass documentation, and com back. Stay tuned.

[slate]

I cannot find anything usable in metaclass for that. I am not smart enough maybe.

Why should the Device inherit? The class in device can have an object reference to an attribute object and can delegate the function calls to this magic object. It is a proxy.

Main program:

#select version of pyapi to import for example from 
sys.path.append("path to pyapy current version") #one way to choose a version

import Defines
import Enums
import Structs
import Macros

import Device

magic=Macros.MagicClass()
dev=Device.Device(magic)

Device.py:

class Device(object):
  def __init__(self, magicClassInstance):
    self._magic=magicClassInstance
  def overridden(self,*args):
    #this function wont be called in self._magic
    pass    
  def __getattr__(self, attribute):
    return getattr(self._magic,attribute)

[tzushky]

I've been reading about them... But since it's quite..metaphysical for my level, I am having trouble picturing the direct use for me. I get that they are useful because I could add new functionalities unknown until runtime.... But to make my ClassToInherit a Metaclass...Or even another one if I ever have many kinds of ClassToinherit...I don't know if it might not be too much trouble... or not really what I need since it's the Daughter that needs inexistent information at runtime...

[tzushky]

I've thought about making a Magic instance the device's attribute (in __init__ just make self._magic = Macros.MagicClass() ->same import problem, Macros would not be known here), but I would have had to call in main:
dev._magic.Function()

With your solution, becuase the __getattr__ is overridden, COuld I call in main:

dev.FunctionFromMagic()

directly making the magic instance invisible to the user?If it is so, than this is the way...

[tzushky]

This simple one worked for me, It might just be the trick I needed...Was this what you were saying?...Thanks a lot anyway, you've helped me clear a lot of doubts...

If you agree I'll mark it solved, If there's still trouble let me know....

class Magic:
    
    def one(self):
        print 'one'
    def two(self):
        print 'two'

class Device:
    
    def __init__(self, i):
        self.I = i
        
    def mine(self):
        print 'dev\'s'
        

    def __getattr__(self, attribute):
        return getattr(self._magic,attribute)
    
    def __setattr__(self,name,value):
        self.__dict__[name] = value

#Main - consider the two classes defined where I was saying and the imports for them done #in main    
m = Magic()
d = Device(4)
d.__setattr__('_magic',m)

d.mine()
d.one()

[tzushky]

But the problem with the Device function needing to appear in Magic stays....:(

[slate]

There are two issues here
First, a proxy instance (device) that delegates unoverridden attributes to the magic instance.
Second, the magic instance should execute functions in device instance.

Is this so?

I dont know the problem, but it seems that the Magic instance cannot work without the device instance. If so, than you could call the device instance directly from the magic code. It does not matter, that the magic.device does not exists on code parsing time.

class Magic:
    def add_device(self,device):
        self.device=device
    def dosomething(self, arg):
       #code
       self.device.function(arg1,arg2)
       #more code

I don't think I can say more without knowing the architecture and the problem.

[tzushky]

It was what I was imagining would remain for a solution, you've understood the situation very well... I honestly will think more about what to choose, this circularity between the two classes is quite...well, it needs very careful handling...

I'd like to thank you very much for your time and help, putting heads together got things fast on track...I'll close the thread because I don't think much more can be said for now...