[Python] The stumbling block of import

A personal note about Python imports. If you have any mistakes or other stumbling blocks, please comment and PR.

Implicit relative import cannot be used for import in package

The story after Python3.

For example

$ tree .
.
└── mypackage
    ├── __init__.py
    ├── mymodule1.py
    └── mymodule2.py

I want to import mymodule1.py from mymodule2.py when the directory structure is.

python:./mypackage/mymodule1.py

A = 1000

def show():
    print('A: {}'.format(A))

python:./mypackage/mymodule2.py

import mymodule1

B = 100

def show():
    print('A: {}, B: {}'.format(mymodule1.A, B))

When,

$ python2.7
Python 2.7.10 (default, Oct 23 2015, 19:19:21)
[GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import mypackage.mymodule2 as mymod2
>>> mymod2.show()
A: 1000, B: 100

$ python3
Python 3.5.2 (default, Aug  4 2016, 09:38:15)
[GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import mypackage.mymodule2 as mymod2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/yusuke-nishioka/tests/tests1/mypackage/mymodule2.py", line 3, in <module>
    import mymodule1
ImportError: No module named 'mymodule1'

Like, in Python 2.7, mymodule1.py can be imported from mymodule2.py, You can see that it cannot be imported in Python 3.5.

Cause

ʻImport mymodule1in mymodule2.py It is intended to import mymodule1.py in the same hierarchy (this is called implicit relative import). However, if there is a package with the same name insys.path`, Are you trying to import mypackage / mymodule1.py? Are you trying to import a package called mymodule on sys.path? There is a problem that it cannot be distinguished.

Therefore, in Python3 series, ʻimport xxx is changed to import the package existing in sys.path` (called absolute import). Implicit relative import is no longer possible.

solution

To import the modules in the package It must be clearly stated that it is a relative import, such as from .import mymodule1 (this is called explict relative import).

Reference URL

https://www.python.org/dev/peps/pep-0328/#rationale-for-relative-imports

When the __package__ attribute is None and the __name__ attribute is __main__, relative import cannot be used.

Well, I mentioned above that I use explicit relavtive import to import the modules in the package, There are cases where relative import cannot be used even though the module is in the package. That is when the __package__ attribute of the module in the package is None and the __name__ attribute is __main__.

$ tree .
.
└── mypackage
    ├── __init__.py
    ├── main.py
    └── mymodule.py

mypackge/mymodule.py

A = 1000

def show():
    print('A: {}'.format(A))

mypackage/main.py

from . import mymodule

if __name__ == '__main__':
    print('__package__: {}, __name__: {}'.format(
        __package__, __name__))
    mymodule.show()

When you run main.py

$ python3 mypackage/main.py
Traceback (most recent call last):
  File "mypackage/main.py", line 1, in <module>
    from . import mymodule
ImportError: cannot import name 'mymodule'

Is displayed, and you can see that mypackage / mymodule.py could not be imported.

Cause

At this time, the __package attribute of main.py is None, and the __name__ attribute is __main__. It is recognized as a top-level module. Therefore, . of from . becomes main.py itself and mymodule cannot be imported.

Solution 1

Let from .import mymodule be from mymodule import show.

Solution 2

Execute with the -m option.

$ python3 -m mypackage.main
__package__: mypackage, __name__: __main__
A: 1000

The __package attribute is mypackage and the __name__ attribute is __main__ . Of from . will point to the same hierarchy as main.py, You can import mymodule.py with from .import mymodule.

By the way, the execution of main.py by -m must be done from outside the package.

$ cd mypackage
$ python3 -m main
Traceback (most recent call last):
  File "/Users/yusuke-nishioka/.anyenv/envs/pyenv/versions/3.6.1/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Users/yusuke-nishioka/.anyenv/envs/pyenv/versions/3.6.1/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/yusuke-nishioka/Documents/United/tmp/mypackage/main.py", line 1, in <module>
    from . import mymodule
ImportError: attempted relative import with no known parent package

Reference URL

https://www.python.org/dev/peps/pep-0328/#relative-imports-and-name

You can import packages without __init__.py

The story of Python 3.3 or later.

$ tree
.
├── mypackage1
│   ├── __init__.py
│   └── subdir1
│       ├── __init__.py.bak
│       └── mymodule1.py
└── mypackage2
    └── subdir1
        └── mymodule2.py

When,

$ python3
Python 3.5.2 (default, Aug  4 2016, 09:38:15)
[GCC 4.2.1 Compatible Apple LLVM 7.3.0 (clang-703.0.31)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import mypackage1
>>> import mypackage2
>>> dir(mypackage1)
['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
>>> dir(mypackage2)
['__doc__', '__loader__', '__name__', '__package__', '__path__', '__spec__']

Like, you can import without __init__.py. Regular package, mypackage1 with __init__.py Mypackage2 without __init__.py is called a namespace package.

further,

>>> import sys
>>> sys.path.append('./mypackage1')
>>> sys.path.append('./mypackage2')
>>> import subdir1
>>> dir(subdir1)
['__doc__', '__loader__', '__name__', '__package__', '__path__', '__spec__']
>>> subdir1.__path__
_NamespacePath(['./mypackage1/subdir1', './mypackage2/subdir1'])

As a package that belongs to the same namespace even if the directory has a different path but the same name Stored in the _NamespacePath object.

Import order

When ʻimport xxx` is executed,

• If xxx / __ init__.py exists in sys.path, it can be obtained as a regular package.
• If xxx / __ init __. Py does not exist in sys.path but xxx. {Py, pyc, so} exists, it can be obtained as a module.
• If neither xxx / __ init __. py norxxx. {Py, pyc, so}exists in sys.path but a directory with the same name exists, it can be obtained as a namespace package.

Difference between regular package and namespace package

The difference is

• namespace package does not have __file__ attribute
• The regular package __path __ is a list, but the namespace package __path __ is a _NamespacePath object.

and so on.

In addition, by making it a namespace package (eliminating __init__.py),

>>> import subdir1.mymodule1
>>> import subdir1.mymodule2
>>> subdir1.mymodule1.__file__
'./mypackage1/subdir1/mymodule1.py'
>>> subdir1.mymodule2.__file__
'./mypackage2/subdir1/mymodule2.py'

With mymodule1.py under ./mypackage1/subdir1, like It can be imported separately from mymodule2.py under ./mypackage2/subdir1.

Summary

I can't think of a scene where I intentionally use the namespace package, If you inadvertently forget __init__.py, you can import it as a module. However, considering that the initialization process can be executed at the time of import (such as setting __all__), Unless you have a specific reason, it seems better to put __init__.py.

See PEP 420 --Implicit Namespace Packages for more information.

Reference URL

PEP 420 -- Implicit Namespace Packages 2012/07/24 I tried the namespace package of Python 3.3b1