Lazily import names in a module

Updated Gribouillis 1 Tallied Votes 468 Views Share

This snippet allows one to implement an equivalent of a __getattr__() function in a python module. The module can then define its own way to lazily import abitrary symbols. For example the module can contain infinitely many names or load values from other modules on demand.

There are many implementations of lazy imports in python, this is a simple one which doesn't interfere with python's import mechanism.

#!/usr/bin/env python
# -*-coding: utf8-*-
# Title: lazynames.py
# Author: Gribouillis for the python forum at www.daniweb.com
# Created: 2012-09-25 12:20:15.556784 (isoformat date)
# License: Public Domain
# Use this code freely.

""" module: lazynames -

    Usage:

        This module allows a python module to import symbols lazily
        by declaring a "name getter function" like module A below:
            
            # module A
            import lazynames

            @lazynames.set_getter(__name__)
            def func(module, attribute):
                return attribute.upper()
        
        The client module B below imports symbols from A which
        are not defined in A 
        
            # module B
            from A import foo, bar, baz
            print(foo, bar, baz) # prints FOO BAR BAZ
            
        When foo, bar, baz are imported, module A uses its
        "name getter function" to create values for these names.
        Optionaly, the values can be stored in module A, so that
        they would be created only once:
            
            @lazynames.set_getter(__name__)
            def func(module, attribute):
                value = attribute.upper()
                setattr(module, attribute, value)
                return value
        
        The following operations are possible:
            
            lazynames.set_getter("my.mod")(func)
                Sets a name getter function for the module named "my.mod".
                Module my.mod must already exist in sys.modules for this
                to work.
                This can be used more than once for the same module.
                lazynames.set_getter("my.mod") can also be used as a
                function decorator as shown above.
                The "name getter function" can be called __getattr__ as
                it behaves like a __getattr__ function: an expression
                like my.mod.foo calls the name getter function with
                argument "foo" if the name "foo" does not already exist
                in module my.mod. The name getter function should raise
                AttributeError to indicate that it can not create a value
                for a given name.
                By default, set_getter won't set a name getter for the
                __main__ module, unless a special flag is passed:
                    lazynames.set_getter("__main__", allow_main = True)(func)
                This means that if the example module A above is used
                as a script instead of being imported, the name getter
                won't be set.
            
            lazynames.get_getter("my.mod")
                returns the current name getter function of module my.mod
                if it has been defined, otherwise return None

"""

from collections import namedtuple
from functools import partial
import sys

__all__ = ["get_getter", "set_getter", "LazyModule"]

Record = namedtuple("Record", "module getter")
REGISTERED = dict()

class LazyModule(object):
    def __init__(self, dict):
        self.__dict__ = dict
        
    def __getattr__(self, attr):
        record = REGISTERED[self.__name__]
        try:
            return getattr(record.module, attr)
        except AttributeError:
            if attr.startswith("__"):
                raise
            else:
                return record.getter(self, attr)

def get_getter(name):
    return REGISTERED[name].getter if name in REGISTERED else None

def set_getter(name, allow_main = False):
    return partial(_register, name = name, allow_main = allow_main)

def _register(getter_func, name = "__main__", allow_main = False):
    if getter_func is None:
        _unregister(name)
    elif allow_main or name != "__main__":
        mod = (REGISTERED[name].module if name in REGISTERED
                else sys.modules[name])
        record = Record(module = mod, getter = getter_func)
        REGISTERED[name] = record
        sys.modules[name] = LazyModule(mod.__dict__)
    return getter_func

def _unregister(name):
    try:
        record = REGISTERED.pop(name)
    except KeyError:
        pass
    else:
        sys.modules[name] = record.module
TrustyTony 888 pyMod Team Colleague Featured Poster

It is, I think, necessary why would somebody to consider this kind of complication to use. One reason is that the importing of large modules takes considerable time, so it would be nice to delay the import itself to time when the function from module is actually called, so execution of importer could continue without being paused waiting the import to complete before continueing (to see example of problem install sympy and see how long it takes to do import from command line).

While doing Google on the topic I found this interesting concept: http://sayspy.blogspot.fi/2009/06/lazy-imports-in-25-lines-of-code.html

Gribouillis 1,391 Programming Explorer Team Colleague

There may be other reasons than importing large modules. An old example is the module Pmw where a lazy importer selects between different versions of the module at run time and imports widgets classes only when they are needed. A module like this can be used as a "facade" to a whole collection of modules and methods.

Also, redefining __getattr__() has many applications in python programs. Why not do it for modules ? (perhaps it already exists in python 3 ?)

Be a part of the DaniWeb community

We're a friendly, industry-focused community of developers, IT pros, digital marketers, and technology enthusiasts meeting, networking, learning, and sharing knowledge.