getting object definitions from a restricted namespace

[Tcll]

basically, I have some code defined to be executed in a restricted namespace:

code = '''
def func():
    print 'success'
'''

NS = {}
NS.update(__builtins__)

exec code in NS

try:
    f = NS['func']
    print f # should print a function object
    f()
except KeyError: print 'failure'

as you can guess... it fails
how can I make it succeed??

really what I'm trying to do is collect the function definition for intellisense operations in my IDE.
(basically I'm doing what other python IDEs that boast about intellisense (except VS2010 with Python Tools) fail to do)

[Gribouillis]

I only removed line 7 and it worked.

Edit: You can also use this initialization

NS = {'__builtins__': __builtins__}

[Tcll]

with that, my intellisense database has no info about builtins (None and int() post a "No Data" tooltip)
screenshot with the working builtins code above: http://lh3.ggpht.com/-1AO-B_70zms/Vd0-RFck_pI/AAAAAAAAJqs/hrYaHZXVZH8/s1600-Ic42/SIDE_intellisense_builtins.png
granted it's not really needed because it's already included in the dict during execution (just not returned).

either way it's the same issue...
the namespace is restricted so I can provide my backend to it while limiting the access and functions provided.

I guess a better example would be:

code = '''
header()

def func():
    print 'success'
'''

def dummy_header(): pass
NS = {'header': dummy_header}
#NS.update(__builtins__) # this will be included once the script code returns it's dict.

exec code in NS
try:
    f = NS['func']
    print f # should print a function object
    f()
except KeyError: print 'failure'

EDIT:
if needed, here's my very basic intellisense collector (it will majorly improve)

def collect(this): #intellisense
    this.database = {}
    for toktype, toktext, (srow, scol), (erow, ecol), line in this.tokens:
        if toktype == tokenize.NAME:
            if toktext in this.keywords: continue
            """
            if toktext in this.moddict:
                print toktext, ':', this.moddict[toktext]
            else:
                print toktext
            #"""
            offset = len(''.join(this.lines[:srow-1]))+scol
            for i in xrange(toktext.__len__()):
                if toktext in this.moddict:
                    this.database[offset+i] = str(this.moddict[toktext])
                else:
                    this.database[offset+i] = "No Data"

this.moddict == NS from above

[Gribouillis]

You could try

NS.update(vars(__builtins__))

[Tcll]

nah, that's not exactly the problem...
I mean, it gets the builtins working yes, but I'm still left without any definitions from the execution code.

if I can get those definitions, I won't need to brute-force definition of the builtins

so yea... any idea how I gan get the definitions while supplying a filled dict??

or perhaps I could exec twice:

NS = {}
try: exec code in NS # fill the namespace
except: pass # because header() is not defined
NS['header'] = dummy_header
exec code in NS # update the rest of the dict with the provided backend namespace (not shown here)

would that work??

EDIT:
as of that last line, basically:

ugeScriptType( UGE_MODEL_SCRIPT ) # set the namespace (gives the script it's particular functions)

where ugeScriptType would be dummy_header from above

[Gribouillis]

The problem is that your code works very well for me. I get

<function func at 0x7f48097fd230>
success

so I don't understand the issue.

[Tcll]

the issue is this:
http://lh3.ggpht.com/-1NwZdA4R4g0/Vd3dMYkvhsI/AAAAAAAAJrE/DEj22hFEdwI/s1152-Ic42/SIDE_intellisense_problem.png
^ the tooltip should be saying "None" for the _GAME variable
but the data isn't returned from the execution:

def execute(this):
    this.moddict = {}

    try: exec this.data in this.moddict # update intellisense database (not working)
    except: pass # there WILL be errors

    this.moddict.update(this.basemoddict)
    #try:
    exec this.data in this.moddict # update intellisense database (not working)
    #except: pass # ignore all errors


    this.lines = this.data.splitlines(True)
    try: this.tokens = list(tokenize.generate_tokens( iter(this.lines).next ))
    except: pass
    this.numLines = len(this.lines) # set defaults
    this.addrlen = len(this.numLines.__str__())+3
    this.textWidth = max([len(l) for l in this.lines])

sorry I didn't have the mentality to state this earlier.
lol

[Gribouillis]

Replace all the except: pass with except: raise and you'll have a detailed report about what's going wrong.

You could also log the exception tracebacks to files if you prefer.

[Tcll]

the error there is known because the header functions are undefined.
that's what this.moddict.update(this.basemoddict) is for

but the problem is this.moddict still doesn't contain the definitions in the script, thus the database displays the tooltip:
http://lh3.ggpht.com/-1NwZdA4R4g0/Vd3dMYkvhsI/AAAAAAAAJrE/DEj22hFEdwI/s1152-Ic42/SIDE_intellisense_problem.png

how do I get the definitions from the script??

[Tcll]

screw it... feel free to download and test:
https://copy.com/DB24NZi4ejlxFo4I
(run SIDE.bat on windows, or SIDE_wine.desktop on linux)

the intellisense database is only updated with the backend definitions,
so that's the only tooltips that will display anything other than "No Data"

however the function definitions in the script for def ugeImportModel(T,C): actually work >_>
but nothing else... None returns "No Data"

EDIT:
to test, open one of UMC's scripts and hover your mouse over any variable, function, or class.
if it works correctly it should display the object referred to in the python backend.

[Tcll]

update:
just did this test:

def execute(this):
    NS = {}
    try: exec this.data in NS # update intellisense database (not working)
    except: pass # there WILL be errors
    print NS

    this.moddict = NS
    this.moddict.update(this.basemoddict)
    #try:
    exec this.data in this.moddict # update intellisense database
    #except: pass # ignore all errors

NS only has 1 key being '__builtins__'
it does not contain anything defined in the script

EDIT:
updated execute() to just 1 exec, since the initial doesn't work:

def execute(this):
    this.moddict = this.basemoddict.copy()
    #try:
    exec this.data in this.moddict # update intellisense database (not working: script definitions not in this.moddict)
    #except: pass # ignore all errors

    this.lines = this.data.splitlines(True)
    try: this.tokens = list(tokenize.generate_tokens( iter(this.lines).next ))
    except: pass
    this.numLines = len(this.lines) # set defaults
    this.addrlen = len(this.numLines.__str__())+3
    this.textWidth = max([len(l) for l in this.lines])

I also re-enabled the builtins from my initial code (these show up in the database once again)

[Tcll]

I think I know what might be going on

basically it's dealing with the functions defined in the script, such as:

def ugeImportModel( T, C ):

that's in the intellisense database as I'd stated earlier, however, it's contents are not...
so it looks like I need to play with the database['ugeImportModel'].__code__ object to get the co_varnames and co_consts of the function, as tested here:

>>> def f(FT,CMD):
    def test(obj): t=3; return obj
    base = 50
>>> f.__code__.co_varnames
('FT', 'CMD', 'test', 'base')
>>> f.__code__.co_consts
(None, <code object test at 0099B338, file "<pyshell#17>", line 2>, 50)
>>> 

one problem though:

>>> dict(zip(f.__code__.co_varnames,f.__code__.co_consts))
{'test': 50, 'CMD': <code object test at 0099B338, file "<pyshell#17>", line 2>, 'FT': None}

looks like I'd need to use inspect? to properly link everything up...
any idea how I could do this??

[Tcll]

here's an interactive example that fully describes my problem:

>>> code = '''
def f(FT,CMD):
    def test(obj): t=3; return obj
    base = 50
'''

>>> ns = {}
>>> exec code in ns
>>> ns['base']

Traceback (most recent call last):
  File "<pyshell#55>", line 1, in <module>
    ns['base']
KeyError: 'base'
>>> ns['int']

Traceback (most recent call last):
  File "<pyshell#56>", line 1, in <module>
    ns['int']
KeyError: 'int'
>>> 

[~s.o.s~]

The first one fails because base is inside the function which only comes into the picture when the function is executed. When you exec the code string, the top level definitions get executed, the functions themselves don't.

The second one fails because 'int' is not directly present in the ns dict, it's part of ns['__builtins__']['int']. If you really need access to int at the top-most level, you can start off your namespace by something like ns2 = dict( __builtins__.__dict__.iteritems() ).

If you are really trying to write a "intellisense" like thing, your best bet (more like a safer bet) would be to use parsing tools to parse the python code and create meaningful representation out of it (just like all other basic intellisense editors out there :) ). Something like this:

import ast

code='''
def foo():
    def bar():
        pass
    myval = 10
'''
res = ast.parse(code)
res.body[0].name # your top level function name i.e. foo
res.body[0].body[1].targets[0].id # function local variable i.e. myval
res.body[0].body[1].value.n  # function local variable's value i.e. 10

This can get hairy very quick so I'm sure there might be a few open source wrappers lying around for this sort of parsing...

[Tcll]

thanks man, I'll start looking into this stuff :)

only reason I'm using exec is for the 2 areas on the right (you can only see one) which are filled with file data and analytics which coincide with the script on the left that operates on the file, so intellisense needs to play a part in that as well. ;)

[Tcll]

aww I liked the look of daniweb, now they made it look stupid :(

anyways...
@~s.o.s~: I can't seem to find any of these related wrappers, or google's just being stupid as usual... 9_9
could you reference me one, and I can find relations if I need better plz. :)

EDIT:
also, ast can't give me the name of a function call, it just reports a call operator... heh

I'm not too sure though how intellisense works with that stuff >_>
so bleh... lol

[Tcll]

EDIT2:
nvm, I just figured it out:

>>> import ast
>>> code = '''
header()

def func(T,C):

    def test(offset):
        return offset

    var0 = 0
    var1 = bf32()
'''
>>> res = ast.parse(code)
>>> body = res.body
>>> body[0].value.func.id
'header'

so it's all about the identifiers... I see

so I could use ast to give me the sort-of python-objects I'd need, right??
btw, header is used like that to portray it's a backend function of my program...
how could I inject that into the AST interface??
or would I just use the python object created at my IDE's runtime??

[Tcll]

alright so the main purpose of this thread is solved which was to use ast (or something similar) on my code, however the purpose of this thread's creation is far from solved which was ultimately a working code-insight database.

I'll mark this as solved and create a new thread about building a professional intellisense database (hopefully something that others can follow as well).
now that I know that AST's are the trick to intellisense, I know how to ask, so thanks for the help. :)