Importing multiple modules from a subfolder to another folder

[Arman777]

I am working on Windows 10 and using VSCode.

In my project, the folder/file tree looks like this;

\Equations
    __init__.py
    equation_producer.py

\Objects
__init__.py
    \GRTensors
     __init__.py
     metrictensor.py
     riccitensor.py
     riemanntensor.py
      ...

Now I want to import the classes in the metrictensor.py, riccitensor.py, riccitensor.py ... files into equation_producer.py

In this configuration, when I type

from Objects.GRTensors.metrictensor import MetricTensor

In equation_producer.py The code works.

However, the problem is that I have eight more files under \GRTensors. So I need to write something like

from Objects.GRTensors import *

So that I don't have to write 10 imports.

In order to solve the problem, I have tried to create

__all__ = ['metrictensor', 'riccitensor', 'riemanntensor'...]

in \Equations\__init__.py but it did not work (if I did not make a mistake/typo somewhere)

I have also tried to write.

from Objects.GRTensors.metrictensor import MetricTensor
from Objects.GRTensors.ricciscalar import RicciScalar
from Objects.GRTensors.riccitensor import RicciTensor

In \Equations\__init__.py but it did not work as well.

So, how can I import all the classes into \Equations\equation_producer.py?

Note: Solutions that are modifiying __init__.py file are prefered.

 

[PeterDonis]

So that I don't have to write 10 imports.

Unfortunately, you do; there is no way to avoid writing them somewhere to do what you want to do. But you can put them in the __init__.py file in the GRTensors folder, and then from Objects.GRTensors import * will work in Equations/equation_producer.py (or anywhere else).

 

[Arman777]

Unfortunately, you do; there is no way to avoid writing them somewhere to do what you want to do. But you can put them in the __init__.py file in the GRTensors folder, and then from Objects.GRTensors import * will work in Equations/equation_producer.py (or anywhere else).

Thanks for your answer. I have realized that I was putting __add__ in wrong __init__.py file.
I just did make the code work. So as you have said I added __add__ in \GRTensor__init_.py and then, when I type

from Objects.GRTensors import *

in Equations/equation_producer.py , I can use the classes by typing

metrictensor.MetricTensor()
christoffelsymbol.ChristoffelSymbol()
etc

Do you think is there way to import it in such a way that I can just type MetricTensor(), in order to run the code ?

 

[PeterDonis]

I have realized that I was putting __add__ in wrong __init__.py file.

I think you mean __all__.

in Equations/equation_producer.py , I can use the classes by typing

metrictensor.MetricTensor()
christoffelsymbol.ChristoffelSymbol()
etc

Do you think is there way to import it in such a way that I can just type MetricTensor(), in order to run the code ?

Instead of using __all__ in Objects/GRTensors/__init__.py the way you are doing it, you would need to write all 10 individual imports there, i.e., from metrictensor import *, from christoffelsymbol import *, etc. Then, in Equations/equation_producer.py, you could just do from Objects.GRTensors import * and you would have MetricTensor, etc., directly available.

 

[Arman777]

I think you mean __all__.

Yes sorry.

I think you mean __all__.Instead of using __all__ in Objects/GRTensors/__init__.py the way you are doing it, you would need to write all 10 individual imports there, i.e., from metrictensor import *, from christoffelsymbol import *, etc. Then, in Equations/equation_producer.py, you could just do from Objects.GRTensors import * and you would have MetricTensor, etc., directly available.

It works thanks a lot :)

 

[PeterDonis]

It works thanks a lot :)

You're welcome, glad it worked!

 

[Arman777]

@PeterDonis

Hey again. I told you that the code worked...actually I found that it did not...(sounds strange but true).

After that I have made a lot of reasearch. Finally, I have came across with this site , which really helped me to understand how the import works.

I have realized that I don't need any __ini__.py file to import functions/classes between files (at least for my case). Since I am using VSCode in Windows, its mostly different from other system/IDE's such as PyCharm or Linux or Mac. In any case, instead of writing

from christoffelsymbol import *

in \Objects\GRTensors\__init__.py, I had to write

from Objects.GRTensors.christoffelsymbol import ChristoffelSymbol

in order to run the code.

 

[PeterDonis]

I have realized that I don't need any __ini__.py file to import functions/classes between files

I'm not sure what you mean by this. You certainly do need to have an __init__.py file in the Objects and Objects/GRTensors folders for things to work. But you don't need to do any imports in those files if you prefer doing it the other way you describe.

Since I am using VSCode in Windows, its mostly different from other system/IDE's such as PyCharm or Linux or Mac.

What kind of IDE you are using is irrelevant to how you do your imports. Your Python code does not know or care what IDE you are using to write it.

instead of writing

from christoffelsymbol import *

in \Objects\GRTensors\__init__.py, I had to write

from Objects.GRTensors.christoffelsymbol import ChristoffelSymbol

in order to run the code.

You could also do

from .christoffelsymbol import *

in Objects\GRTensors\__init__.py, and then

from Objects.GRTensors import *

in Equations/equation_producer.py. Note the dot in front of "christoffelsymbol"; that makes it a relative import. I forgot to put the dot in before; sorry about that.

 

[Arman777]

You certainly do need to have an __init__.py file in the Objects and Objects/GRTensors folders for things to work.

Beforehand I was literally putting __init__.py everywhere. Now my code structure looks like this.

\Equations
    equation_producer.py
\Objects
    \GRTensors
         __init__.py
         metrictensor.py
         riccitensor.py
         riemanntensor.py
     ...
start.py

In this structure, there is

from Objects.GRTensors.christoffelsymbol import ChristoffelSymbol

in Objects\GRTensors\__init__.py

You certainly do need to have an __init__.py file in the Objects

It seems that I don't because the code runs without giving an error (it may be strange, Idk). My whole program is running under the start.py and other then Objects\GRTensors\__init__.py I have not use any __init__.py

In general, I had to use absolute imports everywhere to make the code work. For instance,

riccitensor.py need a class definition from riemanntensor.py and to make the code work I had to write

from Objects.GRTensors.riemanntensor import RiemannTensor

What kind of IDE you are using is irrelevant to how you do your imports. Your Python code does not know or care what IDE you are using to write it.

Indeed, I was just referring to something that I have read in the article that I have give above. If you have to use PYTHONPATH somehow to do imports then it may depend on operating system or IDE how to set your PYTHONPATH

In between the options, you can see, for example, "Add content roots to PYTHONPATH". This is what makes the imports work out of the box when you are in Pycharm but if you run the same code directly from the terminal may give you some issues. You can also edit the environment variables if you click on the small icon to the right of where it says "environment variables".

Another option is to modify the PYTHONPATH environment variable. Environment variables are available on every operating system, the only difference is how you can set and modify them.

It seemed to me that in PyCharm it may differ how we import the code. I have seen lot of PYTHONPATH options in import stuff. I don't know exactly why it's important, but to me it seemed to me that it may change some things.

In any case I don't need any PYTHONPATH stuff so there's no problem on my side.

from .christoffelsymbol import *

this also works yes. Thanks :)

 

[PeterDonis]

It seems that I don't

As long as you don't have any modules at all in the Objects folder itself (i.e., all that is there is subfolders like GRTensors), so you never have to import anything from Objects, yes, this will work, since Python has supported implicit namespace packages since version 3.3:

https://www.python.org/dev/peps/pep-0420/

So what Python is doing is interpreting "Objects" as a namespace package, because that folder doesn't have an __init__.py but it has subfolders that do.

However, if you wanted to import anything from Objects, having it as a namespace package wouldn't work; you would have to add an __init__.py to it to make it a regular package. (You wouldn't have to change anything about GRTensors; regular packages can have sub-packages.)

 

[Arman777]

As long as you don't have any modules at all in the Objects folder itself (i.e., all that is there is subfolders like GRTensors), so you never have to import anything from Objects, yes, this will work, since Python has supported implicit namespace packages since version 3.3:

https://www.python.org/dev/peps/pep-0420/

So what Python is doing is interpreting "Objects" as a namespace package, because that folder doesn't have an __init__.py but it has subfolders that do.

However, if you wanted to import anything from Objects, having it as a namespace package wouldn't work; you would have to add an __init__.py to it to make it a regular package. (You wouldn't have to change anything about GRTensors; regular packages can have sub-packages.)

Hmm that's strange because I have exactly that kind of style...

Let me share a picture that shows some part of my folder style.

as you can see there is simplifyobjects.py under the Objects Folder. In all of the files under GRTensors, I am importing simplifyobjects.py via

from Objects.simplifyobjects import Simplify

without any problem.

If you are saying that if I try to import simplifyobjects.py from another file in different folder (such as from Equations\GRTensor_equation_producer.py) then I don't know the code will work or not.

 

[PeterDonis]

I was just referring to something that I have read in the article that I have give above. If you have to use PYTHONPATH somehow to do imports then it may depend on operating system or IDE how to set your PYTHONPATH

In between the options, you can see, for example, "Add content roots to PYTHONPATH". This is what makes the imports work out of the box when you are in Pycharm but if you run the same code directly from the terminal may give you some issues. You can also edit the environment variables if you click on the small icon to the right of where it says "environment variables".

Another option is to modify the PYTHONPATH environment variable. Environment variables are available on every operating system, the only difference is how you can set and modify them.

It seemed to me that in PyCharm it may differ how we import the code. I have seen lot of PYTHONPATH options in import stuff. I don't know exactly why it's important, but to me it seemed to me that it may change some things.

PyCharm's "Add content roots to PYTHONPATH" setting just modifies the PYTHONPATH environment variable before it runs the Python interpreter. Doing it that way only modifies PYTHONPATH for the interpreter that runs inside PyCharm, since PyCharm is running the interpreter as a subprocess. If, OTOH, you set PYTHONPATH through the operating system, it will affect every instance of the interpreter you run, no matter how you run it (from PyCharm, from the Start menu, from the command line, etc.). None of this changes how the interpreter itself imports code; it just looks in each of the folders in sys.path, in order, until it finds an importable module or package matching the name you are trying to import. All setting PYTHONPATH does is add the folders in PYTHONPATH to sys.path, after the current folder (which is represented in sys.path as an empty string, and will be the first item in the list) but before any other folders (such as the folder where the Python standard library is located).

The article you referenced gives some incorrect information, btw. It says:

Python will first look at whether what you are trying to import belongs to the standard library. This means that if we would have called the folder time instead of numpy, the behavior would have been different.

If Python doesn't find the module in its standard library, it will check for external modules. In it does this also in a very special order. It will first start by searching in the current directory, and then it will move to the directories where packages are installed

This is not correct. Python only does one thing when looking for imports: as I said above, it looks in each directory in sys.path, in order. Since the interpreter by default puts the current directory first in sys.path (represented by an empty string, as noted above), Python will look in the current directory before it looks anywhere else, including the standard library.

Also, since sys.path can be modified at runtime, you can not only add directories to be searched for imports, you can also remove them or re-order them. sys.path is just an ordinary Python list and can be manipulated like any other list, and if you do that before an import statement, that import will use the modified sys.path.

 

[PeterDonis]

as you can see there is simplifyobjects.py under the Objects Folder. In all of the files under GRTensors, I am importing simplifyobjects.py via

from Objects.simplifyobjects import Simplify

without any problem.

Hm, interesting. That means namespace packages are more general than I had thought.

If you are saying that if I try to import simplifyobjects.py from another file in different folder (such as from Equations\GRTensor_equation_producer.py) then I don't know the code will work or not.

It should work the same as you describe above.

 

[PeterDonis]

The Python documentation on the import system is worth reading:

https://docs.python.org/3/reference/import.html

 

[Arman777]

Thanks