Python|Import机制备忘-模块搜索路径(sys.path)、嵌套Import、package Import

Python Import机制备忘-模块搜索路径(sys.path)、嵌套Import、package Import　2009-03-11 15:09

分类：python

字号： 大大  中中  小小

模块的搜索路径

模块的搜索路径都放在了sys.path列表中，如果缺省的sys.path中没有含有自己的模块或包的路径，可以动态的加入（sys.path.apend）即可。下面是sys.path在Windows平台下的添加规则。

1、sys.path第一个路径往往是主模块所在的目录。在交互环境下添加一个空项，它对应当前目录。

2、如果PYTHONPATH环境变量存在，sys.path会加载此变量指定的目录。

3、我们尝试找到Python Home，如果设置了PYTHONHOME环境变量，我们认为这就是Python Home，否则，我们使用python.exe所在目录找到lib\os.py去推断Python Home。

如果我们确实找到了Python Home，则相关的子目录（Lib、plat-win、lib-tk等）将以Python Home为基础加入到sys.path，并导入（执行）lib/site.py，将site-specific目录及其下的包加入。

如果我们没有找到Python Home，则把注册表Software\Python\PythonCore\2.5\PythonPath的项加入sys.path（HKLM和 HKCU合并后加入），但相关的子目录不会自动添加的。

4、如果我们没有找到Python Home，并且没有PYTHONPATH环境变量，并且不能在注册表中找到PythonPath，那么缺省相对路径将加入（如：.\Lib;.\plat-win等）。

总结如下

当在安装好的主目录中运行Python.exe时，首先推断Python Home，如果找到了PythonHome，注册表中的PythonPath将被忽略；否则将注册表的PythonPath加入。

如果PYTHONPATH环境变量存在，sys.path肯定会加载此变量指定的目录。

如果Python.exe在另外的一个目录下（不同的目录，比如通过COM嵌入到其他程序），Python Home将不推断，此时注册表的PythonPath将被使用。

如果Python.exe不能发现他的主目录（PythonHome），并且注册表也没有PythonPath，则将加入缺省的相对目录。

 

标准Import

Python中所有加载到内存的模块都放在sys.modules。当import一个模块时首先会在这个列表中查找是否已经加载了此模块，如果加载了则只是将模块的名字加入到正在调用import的模块的Local名字空间中。如果没有加载则从sys.path目录中按照模块名称查找模块文件，模块文件可以是py、pyc、pyd，找到后将模块载入内存，并加入到sys.modules中，并将名称导入到当前的Local名字空间。

可以看出了，一个模块不会重复载入。多个不同的模块都可以用import引入同一个模块到自己的Local名字空间，其实背后的PyModuleObject对象只有一个。

说一个容易忽略的问题，import只能导入模块，不能导入模块中的对象（类、函数、变量等）。如一个模块A（A.py）中有个函数getName，另一个模块不能通过import A.getName将getName导入到本模块，只能用import A。如果想只导入特定的类、函数、变量则用from A import getName即可。

嵌套Import

嵌套import，我分两种情况，一种是：本模块导入A模块（import A），而A中又有import语句，会激活另一个import动作，如import B，而B模块又可以import其他模块，一直下去。

对这种嵌套比较容易理解，注意一点就是各个模块的Local名字空间是独立的，所以上面的例子，本模块import A完了后本模块只能访问模块A，不能访问B及其他模块。虽然模块B已经加载到内存了，如果要访问还要在明确的在本模块中import B。

另外一种嵌套指，在模块A中import B，而在模块B中import A。这时会怎么样呢？这个在Python列表中由RobertChen给出了详细解释，抄录如下：

[A.py] from B import D class C:pass [B.py] from A import C class D:pass

为什么执行A的时候不能加载D呢？

如果将A.py改为：import B就可以了。

这是怎么回事呢？

RobertChen：这跟Python内部import的机制是有关的，具体到from B import D，Python内部会分成几个步骤：

1. 在sys.modules中查找符号"B"
2. 果符号B存在，则获得符号B对应的module对象<module B>。

   从<module B>的__dict__中获得符号"D"对应的对象，如果"D"不存在，则抛出异常

3. 如果符号B不存在，则创建一个新的module对象<module B>，注意，这时，module对象的__dict__为空。执行B.py中的表达式，填充<module B>的__dict__ 。

   从<module B>的__dict__中获得"D"对应的对象，如果"D"不存在，则抛出异常。

所以，这个例子的执行顺序如下：

1、执行A.py中的from B import D

由于是执行的python A.py，所以在sys.modules中并没有<module B>存在，首先为B.py创建一个module对象(<module B>)，注意，这时创建的这个module对象是空的，里边啥也没有，在Python内部创建了这个module对象之后，就会解析执行B.py，其目的是填充<module B>这个dict。

2、执行B.py中的from A import C

在执行B.py的过程中，会碰到这一句，首先检查sys.modules这个module缓存中是否已经存在<module A>了，由于这时缓存还没有缓存<module A>，所以类似的，Python内部会为A.py创建一个module对象(<module A>)，然后，同样地，执行A.py中的语句。

3、再次执行A.py中的from B import D

这时，由于在第1步时，创建的<module B>对象已经缓存在了sys.modules中，所以直接就得到了<module B>，但是，注意，从整个过程来看，我们知道，这时<module B>还是一个空的对象，里面啥也没有，所以从这个module中获得符号"D"的操作就会抛出异常。如果这里只是import B，由于"B"这个符号在sys.modules中已经存在，所以是不会抛出异常的。

上面的解释已经由Zoom.Quiet收录在啄木鸟了，里面有图，可以参考一下。

Package（包） Import

包（Package）可以看成模块的集合，只要一个文件夹下面有个__init__.py文件，那么这个文件夹就可以看做是一个包。包下面的文件夹还可以成为包（子包）。更进一步，多个较小的包可以聚合成一个较大的包，通过包这种结构，方便了类的管理和维护，也方便了用户的使用。比如SQLAlchemy等都是以包的形式发布给用户的。

包和模块其实是很类似的东西，如果查看包的类型import SQLAlchemy type(SQLAlchemy)，可以看到其实也是<type 'module'>。import包的时候查找的路径也是sys.path。

包导入的过程和模块的基本一致，只是导入包的时候会执行此包目录下的__init__.py而不是模块里面的语句了。另外，如果只是单纯的导入包，而包的__init__.py中又没有明确的其他初始化操作，那么此包下面的模块是不会自动导入的。如：

PA

--__init__.py

--wave.py

--PB1

  --__init__.py

  --pb1_m.py

--PB2

  --__init__.py

  --pb2_m.py

__init__.py都为空，如果有以下程序：

 

1. import sys
2. import PA.wave  #1
3. import PA.PB1   #2
4. import PA.PB1.pb1_m as m1  #3
6. import PA.PB2.pb2_m #4
8. PA.wave.getName() #5
10. m1.getName() #6
12. PA.PB2.pb2_m.getName() #7

当执行#1后，sys.modules会同时存在PA、PA.wave两个模块，此时可以调用PA.wave的任何类或函数了。但不能调用PA.PB1(2)下的任何模块。当前Local中有了PA名字。

当执行#2后，只是将PA.PB1载入内存，sys.modules中会有PA、PA.wave、PA.PB1三个模块，但是PA.PB1下的任何模块都没有自动载入内存，此时如果直接执行PA.PB1.pb1_m.getName()则会出错，因为PA.PB1中并没有pb1_m。当前Local中还是只有PA名字，并没有PA.PB1名字。

当执行#3后，会将PA.PB1下的pb1_m载入内存，sys.modules中会有PA、PA.wave、PA.PB1、PA.PB1.pb1_m四个模块，此时可以执行PA.PB1.pb1_m.getName()了。由于使用了as，当前Local中除了PA名字，另外添加了m1作为PA.PB1.pb1_m的别名。

当执行#4后，会将PA.PB2、PA.PB2.pb2_m载入内存，sys.modules中会有PA、PA.wave、PA.PB1、PA.PB1.pb1_m、PA.PB2、PA.PB2.pb2_m六个模块。当前Local中还是只有PA、m1。

下面的#5，#6，#7都是可以正确运行的。

注意的是：如果PA.PB2.pb2_m想导入PA.PB1.pb1_m、PA.wave是可以直接成功的。最好是采用明确的导入路径，对于./..相对导入路径还是不推荐用。

Python Decorators Don't Have to be (that) Scary - Siafoo

Python Decorators Don't Have to be (that) Scary

By David Updated 9 months ago (30 Jul 2008 at 02:37 PM) history recent activity

Edit Add to Library Subscribe

Send Report

Abstract: Decorators modify functions. Beginning with the basics, learn how to use decorators in a variety of ways. Execute code when a function is parsed or called. Conditionally call functions and transform inputs and outputs. Write customizable decorators that accept arbitrary arguments. And, if necessary, easily make sure your decorated function has the same signature as the original.

Languages Python

Contents

• 1   The Basics
  • 1.1   Does a decorator function have to return a function?
• 2   Run-Time Transformations
  • 2.1   The Wrapper Function
  • 2.2   Getting the Arguments
  • 2.3   Summing It Up
• 3   Dynamic Decorators
  • 3.1   Passing Options to the Decorator
  • 3.2   Run-Time Tranformations
• 4   Caveat: Function Signatures
  • 4.1   A Solution
• 5   Putting it All Together
• 6   Conclusion
• 7   References

Decorators. The shear mention of them brings fear to even the seasoned Python programmer.

Okay, maybe not. But decorators, at least in this author's opinion, have a weird syntax and inevitably complicated implementations that are especially foreign (I think) to many who have gotten used to the simplicity of Python.

1   The Basics

Decorators modify functions. More specifically, a decorator is a function that transforms another function.

When you use a decorator, Python passes the decorated function -- we'll call this the target function -- to the decorator function, and replaces it with the result. Without decorators, it would look something like this:

# 's

 1def decorator_function(target):
   2    # Do something with the target function
   3    target.attribute = 1
   4    return target
  5
 6def target(a,b):
   7    return a + b
   8
 9# This is what the decorator actually does
  10target = decorator_function(target)
  

This code has the exact same functionality, but uses decorators. Note that I can name my decorator function whatever I want. Here, I've chosen 'decorator_function':

# 's

1def decorator_function(target):
  2    # Do something with the target function
  3    target.attribute = 1
  4    return target
 5
6# Here is the decorator, with the syntax '@function_name'
  7@decorator_function
8def target(a,b):
  9    return a + b
  

As you can see, you need to put the decorator function's name, prefaced with a @, on the line before the target function definition. Python internally will transform the target by applying the decorator to it and replacing it with the returned value.

Both of the above examples will have the same results:

# 's

1>>> target(1,2)
  23
3>>> target.attribute
  41

1.1   Does a decorator function have to return a function?

No. The decorator function can return absolutely anything, and Python will replace the target function with that return value. For example, you could do something like this:

# 's

 1def decorator_evil(target):
   2    return False
   3
 4@decorator_evil
   5def target(a,b):
   6    return a + b
   7
 8>>> target
   9False
10
  11>>> target(1,2)
  12TypeError: 'bool' object is not callable
  

This is really not something you want to be doing on a regular basis though -- I'm pretty sure that a basic design principal is to not have functions randomly turning into other sorts of things. It makes good sense to at least return some sort of callable.

2   Run-Time Transformations

"But," I hear you saying, "I thought decorators did more than that. I want to do things at run-time, like conditionally calling the function and transforming the arguments and return value."

Can decorators do these things? Yes. Is that really something 'more' than we talked about above? Not really. It's important here not to get bogged down in the details -- you already know all there is to know about decorators. To do one of these more complex things, we're really just adding some plain old Python to the mix.

2.1   The Wrapper Function

Remember, your decorator function can return an arbitrary function. We'll call it the wrapper function, for reasons which will become clear in a second. The trick here is to define the wrapper function inside the decorator function, giving it access to the decorator function's variable scope, including the target function.

# 's

 1def decorator(target):
   2
 3    def wrapper():
   4        print 'Calling function "%s"' % target.__name__
   5        return target()
   6
 7    # Since the wrapper is replacing the target function, assigning an attribute to the target function won't do anything.
   8    # We need to assign it to the *wrapper function*.
   9    wrapper.attribute = 1
  10    return wrapper
 11
12@decorator
  13def target():
  14    print 'I am the target function'
  15
16>>> target()
  17Calling function "target"
 18I am the target function
 19
20>>> target.attribute
  211

As you can see, the wrapper function can do whatever it wants to the target function, including the simple case of returning the target's return value. But what happens to any arguments passed to the target function?

2.2   Getting the Arguments

Since the returned wrapper function replaces the target function, the wrapper function will receive the arguments intended for the target function. Assuming you want your decorator to work for any target function, your wrapper function then should accept arbitrary non-keyword arguments and arbitrary keyword arguments, add, remove, or modify arguments if necessary, and pass the arguments to the target function.

# 's

 1def decorator(target):
   2
 3    def wrapper(*args, **kwargs):
   4        kwargs.update({'debug': True}) # Edit the keyword arguments -- here, enable debug mode no matter what
   5        print 'Calling function "%s" with arguments %s and keyword arguments %s' % (target.__name__, args, kwargs)
   6        return target(*args, **kwargs)
   7
 8    wrapper.attribute = 1
   9    return wrapper
 10
11@decorator
  12def target(a, b, debug=False):
  13    if debug: print '[Debug] I am the target function'
  14    return a+b
  15
16>>> target(1,2)
  17Calling function "target" with arguments (1, 2) and keyword arguments {'debug': True}
  18[Debug] I am the target function
  193
20
  21>>> target.attribute
  221

Note

You can also apply a decorator to a class method. If your decorator is always going to be used this way, and you need access to the current instance, your wrapper function can assume the first argument is always self:

# 's

1def wrapper(self, *args, **kwargs):
  2    # Do something with 'self'
  3    print self
  4    return target(self, *args, **kwargs)
  

2.3   Summing It Up

So, we have a wrapper function that accepts arbitrary arguments defined inside our decorator function. The wrapper function can call the target function if and when it wants, get the result, do something with it, and return whatever it wants.

Say I want certain function calls to require positive confirmation before they are executed, and then stringify the result of the function before returning it. Note that the built-in function raw_input prints a message and then waits for a response from stdin.

# 's

 1def decorator(target):  # Python passes the target function to the decorator
   2
 3    def wrapper(*args, **kwargs):
   4
 5        choice = raw_input('Are you sure you want to call the function "%s"? ' % target.__name__)
   6
 7        if choice and choice[0].lower() == 'y':
   8            # If input starts with a 'y', call the function with the arguments
   9            result = target(*args, **kwargs)
  10            return str(result)
  11
12        else:
  13            print 'Call to %s cancelled' % target.__name__
  14
15    return wrapper
  16
17@decorator
  18def target(a,b):
  19    return a+b
  20
21>>> test.target(1,2)
  22Are you sure you want to call the function "target"? n
  23Call to target cancelled
 24
25>>> test.target(1,2)
  26Are you sure you want to call the function "target"? y
  27'3'

3   Dynamic Decorators

Sometimes you might want to customize behavior by passing arbitrary options to your decorator function. A cursory look at decorator syntax suggests there's no way to do that. You could just abandon the decorator idea altogether, but you certainly don't have to.

The solution is define your decorator function inside another function -- call it the options function. Right before the target function definition, where you would normally list the decorator function (prepended with an @), call this options function (prepended with an @) instead. The options function then returns your decorator function, which Python will use as the passes the target function to as before.

3.1   Passing Options to the Decorator

Your options function can accept any arguments you want it to. Since the decorator function is defined inside the options function, the decorator function has access to any of the arguments passed to the options function.

# 's

 1def options(value):
   2
 3    def decorator(target):
   4        # Do something with the target function
   5        target.attribute = value
   6        return target
  7    return decorator
  8
 9@options('value')
  10def target(a,b):
  11    return a + b
  12
13>>> target(1,2)
  143
15
  16>>> target.attribute
  17'value'

As you can see, nothing here about the decorator syntax itself has changed. Our decorator function is just in a dynamic scope instead of a static one.

3.2   Run-Time Tranformations

You can do Run-time Transformations by returning a wrapper function from your decorator function, just like before. For better or worse, though, there now must be three levels of functions:

# 's

 1def options(debug_level):
   2
 3    def decorator(target):
   4
 5        def wrapper(*args, **kwargs):
   6            kwargs.update({'debug_level': debug_level}) # Edit the keyword arguments
   7                                                        # here, set debug level to whatever specified in the options
   8
 9            print 'Calling function "%s" with arguments %s and keyword arguments %s' % (target.__name__, args, kwargs)
  10            return target(*args, **kwargs)
  11
12        return wrapper
  13
14    return decorator
  15
16@options(5)
  17def target(a, b, debug_level=0):
  18    if debug_level: print '[Debug Level %s] I am the target function' % debug_level
  19    return a+b
  20
21>>> target(1,2)
  22Calling function "target" with arguments (1, 2) and keyword arguments {'debug_level': 5}
  23[Debug Level 5] I am the target function
  243

4   Caveat: Function Signatures

Phew. Understand everything you can do with decorators now? Good :). However, there is one drawback that must be mentioned.

The function returned from the decorator function -- usually a wrapper function -- replaces the target function completely. Any later introspection into what appears to be the target function will actually be into the wrapper function.

Most of the time, this is okay. Generally you just call a function with some options. Your program doesn't check to see what the function's __name__ or what arguments it accepts. So usually this problem won't be a problem.

However sometimes you care if the function you are calling supports a certain option, supports arbitrary options, or, perhaps, what its __name__ is. Or maybe you are interested in one if the function's attributes. If you look at a function that has been decorated, you will actually be looking at the wrapper function.

In the example below, note that the getargspec function of the inspect module gets the names and default values of a function's arguments.

# 's

 1# This function is the same as the function 'target', except for the name
   2def standalone_function(a,b):
   3    return a+b
   4
 5def decorator(target):
   6
 7    def wrapper(*args, **kwargs):
   8        return target()
   9
10    return wrapper
  11
12@decorator
  13def target(a,b):
  14    return a+b
  15
16>>> from inspect import getargspec
  17
18>>> standalone_function.__name__
  19'standalone_function'
20
  21>>> getargspec(standalone_function)
  22(['a', 'b'], None, None, None)
  23
24>>> target.__name__
  25'wrapper'
26
  27>>> getargspec(target)
  28([], 'args', 'kwargs', None)
  

As you can see, the wrapper function reports that it accepts different arguments than the original target function Its call signature has changed.

4.1   A Solution

This is not an easy problem to solve. The update_wrapper method of the functools module provides a partial solution, copying the __name__ and other attributes from one function to another. But it does not solve what might be the largest problem of all: the changed call signature.

The decorator function of the decorator module provides the best solution: it can wrap your wrapper function in a dynamically-evaluated function with the correct arguments, restoring the original call signature. Similar to the update_wrapper function, it can also update your wrapper function with the __name__ and other attributes from the target function.

Note

For the remainder of this section, when I speak of the decorator function, I mean the one from this module, not one of the decorator functions that we've been using to transform our target functions.

Another way to create decorators

Unfortunately, though, the decorator function wasn't written with this use in mind. Instead it was written to turn standalone wrapper functions into full-fledged decorators, without having to worry about the function nesting described in Run-Time Transformations, above.

While this technique is often useful, it is much less customizable. Everything must be done at run-time, each time the function is executed. You cannot do any work when the target function is defined, including assigning the target or wrapper functions attributes or passing options to the decorator.

Also, in this author's opinion it is a bit of a black box; I'd rather know what my decorators are doing even if it is a little messier.

But we can make it work for us to solve this problem.

Ideally you would just call decorator(wrapper) and be done with it. However, things are never as simple as we'd like. As described above, the decorator function wraps the function passed to it -- our wrapper function -- in a dynamic function to fix the signature. But we still have a few problems:

Problem #1:

The dynamic function calls our wrapper function with (func, *args, **kwargs)

Solution #1:

Make our wrapper function accept (func, *args, **kwargs) instead of just (*args, **kwargs).

Problem #2:

The dynamic function is then wrapped in another function that expects to be used as an actual decorator -- it expects to be called with the target function, and will return the wrapper function.

Solution #2:

Call decorator's return value with the target function to get back to the dynamic function, which has the right signature.

This technique is a bit of a hack, and is a bit hard to explain, but it is easy to implement and works well.

This is the same example as before, but now with the decorator function (and a name change so things don't get too confusing):

# 's

 1from decorator import decorator
   2
 3def my_decorator(target):
   4
 5    def wrapper(target, *args, **kwargs): # the target function has been prepended to the list of arguments
   6        return target(*args, **kwargs)
   7
 8    # We are calling the returned value with the target function to get a 'proper' wrapper function back
   9    return decorator(wrapper)(target)
  10
11
 12@my_decorator
13def target(a,b):
  14    return a+b
  15
16>>> from inspect import getargspec
  17
18>>> target.__name__
  19'target'
20
  21>>> getargspec(target)
  22(['a', 'b'], None, None, None)
  

5   Putting it All Together

Sometimes, you really need a customizable decorator that does work both at parse-time and run-time, and has the signature of the original target function.

Here's an example that ties everything together. Expanding on the example from earlier, say you want certain function calls to require positive confirmation before they are executed, and you want to be able to customize the confirmation string for each target function. Furthermore, for some reason [1], you need the decorated function's signature to match the target function.

Here we go:

# 's

 1from decorator import decorator
   2
 3# The 'options' function.  Recieves options and returns a decorator.
   4def confirm(text):
   5    '''
  6    Pass a string to be sent as a confirmation message.  Returns a decorator.
   7    '''
  8
 9    # The actual decorator.  Recieves the target function.
  10    def my_decorator(target):
  11        # Anything not in the wrapper function is done when the target function is initially parsed
  12
13        # This is okay because the decorator function will copy the attribute to the wrapper function
  14        target.attribute = 1
  15
16        # The wrapper function.  Replaces the target function and receives its arguments
  17        def wrapper(target, *args, **kwargs):
  18            # You could do something with the args or kwargs here
  19
20            choice = raw_input(text)
  21
22            if choice and choice[0].lower() == 'y':
  23                # If input starts with a 'y', call the function with the arguments
  24                result = target(*args, **kwargs)
  25                # You could do something with the result here
  26                return result
  27
28            else:
  29                print 'Call to %s cancelled' % target.__name__
  30
31        # Fix the wrapper's call signature
  32        return decorator(wrapper)(target)
  33
34    return my_decorator
  35
36@confirm('Are you sure you want to add these numbers? ')
  37def target(a,b):
  38    return a+b
  39
40>>> Are you sure you want to add these numbers? yes
  413
42
  43>>> target.attribute
  441

Hey, what do you know, it actually works.

6   Conclusion

As always, if you have a better way to do anything mentioned here, or if I've left anything out, leave a comment or feel free edit this article to fix the problem.

7   References

Michele Simionato's 'decorator' Module

Python Tips, Tricks, and Hacks - Decorators

[1]	For example, Pylons checking your method's signature to decide what arguments to pass it.