Improves documentation about modules management - #17757

Python 3 doesn't need __init__.py's, right, particularly when you're just dealing with a normal dir being added to PYTHONPATH?

Maybe mention __init__.py's and link down to the "Add init.py in package folders" section?

Python 3 doesn't need init.py's, right,

That's not entirely correct... I also got that impression for some time, but (at least I see it), it's a bit of a misunderstanding.
bB
Python 3 can turn regular folders into implicit packages with their own namespaces, But a number of tools discovering python code (airflow DAG loading including) does not deal with implicit namespaces and still requirea an __init__.py files in the folders).

Maybe I am wrong about it ? @ashb ? But that's the impression I have.

I did separate it out, but I would love to have a good statement about it.

Just to add a bit of links on that and my thoughts and context for __init__.py in relation to our DAGs. I've spent quite some time on looking how the packages/namespaces work and while I also had the impression at some point in time that "as of python 3.3 we should drop the __init__,py altogether", my understanding of it is a bit different now.

To focus the discussion - here is the PEP in question: https://www.python.org/dev/peps/pep-0420

And here is particular chapter about impact on import finders and loaders: https://www.python.org/dev/peps/pep-0420/#impact-on-import-finders-and-loaders

My understanding of it:

• lack of __init__.py in the folder means that (providing that import finder and loader supports it), such folder might be detected as "implicit namespace package" and modules could be added to the package in their own namespace. Separate namespaces mean that the same package can be split across multiple folders. But it really depends on the capabilities of loader/importer. For example in setup.py you need to use find_namespace_packages instead of find_packages to find the namespace-packages

• some tools/methods that are looking for packages, might not work well with those namespace packages still (they might expect the package = single folder approach for example and break, or might not find the namespace packages at all). Its less and less common, but it happens and even in some cases the tools be default do not find the namespace packages, you have to explicitly enable it (for example in mypy you need to specify --namespace-packages flag to find them https://mypy.readthedocs.io/en/stable/command_line.html#cmdoption-mypy-namespace-packages

• things can get broken easily if there are same package with classic packages with __init__.py and namespace packages. There are also couple of legacy way to provide namespaces packages (not all compatible with each other) some of those described here https://stackoverflow.com/questions/1675734/how-do-i-create-a-namespace-package-in-python for example but I think there are also some other ways. There are historically ways to modify __init__.py in the way to turn the package into an explicitly namespaced one. But they have some limitations (for example you had to have exactly the same __init__.py content in all the incarnations of the same package in different namespaces. And if you mixed and matched the different ways of defining namespace they might or might not work and behave strangely..

• there is a serious performance implication of using "implicit namespace packages". There is even a note in PEP 420 that there is not even an intention to replace "classic" packages with implicit namespaces:

There is no intention to remove support of regular packages. If a developer knows that her package will never be a portion of a namespace package, then there is a performance advantage to it being a regular package (with an __init__.py). Creation and loading of a regular package can take place immediately when it is located along the path. With namespace packages, all entries in the path must be scanned before the package is created.

I am not even sure if our DAG loader supports implicit namespaces, but even if it does I think we should discourage the use of implicit namespaces for DAGs - if not because of the potential clashes and conflicts with "classic" packages or "explicit namespaces" that people can by accident have on their pythonpath - but also because of performance.

I'd love to hear your thoughts about it :)

Summarising then what this PR is all about in one paragraph (and you can quote me on that and I can add this paragraph to the "modules management"):

Both relative imports and implicit namespaces while somewhat useful features of Python 3, have potential of being misused, misunderstood and break the installation in implicit ways which are difficult to debug and understand. Rather than trying to make them works with Airflow DAG parsing we strongly discourage people from using them. DAGs should be written using absolute imports and regular "classic" packages with __init__.py - for performance reasons, but most of all for the sanity of both the users and committers trying to help them.

😄

Thanks for the detailed response @potiuk, TIL. I knew about (some of) the nuances with find_namespaces_packages vs find_packages, but the performance advantage is a great reason. I'm happy for us to encourage and show examples of "classic" packages. For the sake of sanity is also a really good argument :)