Python Understanding where pip saves the downloaded Python modules

  • Thread starter Thread starter fog37
  • Start date Start date
  • Tags Tags
    Modules Python
AI Thread Summary
The discussion centers on the confusion surrounding the installation paths and usage of multiple Python interpreters on a Windows system, specifically regarding Anaconda and a standard Python installation. The user finds that packages installed via pip are stored in the Anaconda directory, even when using the Python 3.7 interpreter from the standard installation. It is clarified that the active pip corresponds to the Python interpreter being used, which is why modules can be accessed across installations. The conversation also touches on the implications of having both installations, suggesting that keeping one version, preferably Anaconda, may simplify management. Ultimately, the user expresses a desire to resolve the confusion caused by having two Python installations.
fog37
Messages
1,566
Reaction score
108
TL;DR Summary
Understanding which in folder pip saves the downloaded Python modules.
Hello,

I have been trying to figure out where things are stored in Windows. For example, I have two versions of Python, one is stored in the PYthon37 folder and one in the Anaconda3 folder. See below:
1613950803090.png


I have been using pip at the command prompt to install packages/modules and when typing at the command prompt pip list -v, I see that ALL currently installed packages are stored in a subfolder of the anaconda3 folder...I didn't expect because I thought they would we be stored is a subfolder of the Python37 folder instead...
Also, when using the IDLE, which uses the Python3.7 interpreter stored in the python37 folder, I can import numpy and other modules with no problem even if they are stored in the anaconda3 folder...Why? And why did pip store the downloaded modules in a a subfolder of the anaconda3 folder?

1613950600624.png


Thanks!
 
Technology news on Phys.org
Do you also have two pips installed? If so, each is probably configured to install packages into a directory associated with its corresponding Python installation.

If both Pythons are the same version (3.7) then they'll be able to import each other's packages if visible in your PYTHONPATH.
 
  • Like
Likes fog37
at the command prompt, I only see one pip installed in a sub-folder of the anaconda3 folder. So that explains why all the modules are saved in a subfolder of the anaconda3 folder.

1613955802810.png


However, when I am using the IDLE, which was installed with the other Python (not the Anaconda one), the modules are available from the IDLE anyway even if it uses not the Anaconda Python interpreter...

Does it mean that my Python3.7 installation did not come with pip? I thought it would...Should install another pip?
 
There is an IDLE also in anaconda apparently:
1613956974423.png


But when I use the IDLE I launch it from here
1613957152467.png


which is not the IDLE part of anaconda...
 
fog37 said:
I see that ALL currently installed packages are stored in a subfolder of the anaconda3 folder...

I would assume that's because you used the pip that came with the anaconda installation of Python to install those packages. Pip is just a Python program, so whichever Python interpreter is running it will be the one it uses to determine where packages should go.

fog37 said:
when using the IDLE, which uses the Python3.7 interpreter stored in the python37 folder, I can import numpy and other modules with no problem even if they are stored in the anaconda3 folder...Why?

Try looking at sys.path, since that gives the folders that the interpreter will look in for modules and packages.

This is one of those times where "try it!" is a good way of figuring out what is going on.

Btw, is there any particular reason why you have two Python installations? On Linux that's common, because pretty much any Linux install will install whichever version of Python the distro is using for system management scripts, but often that's not the version the user wants to run for their own programs, so they'll install a different one. But Windows doesn't use Python for its own system management, so the only way you'll have a Python installation at all is if you do it yourself.
 
  • Like
Likes fog37
A lot of folks tend to use Anaconda in the Machine Learning field and its corresponding conda multi-purpose command to install packages from the Anaconda repos.

But when you're using the installed pyhtin or just not using the Anaconda distro then folks will use pip to get third party modules. However, Anaconda folks realized new folks coming to Ananconda may still want to use pip and so they provided it in their distro as well.
 
  • Like
Likes fog37
PeterDonis said:
I would assume that's because you used the pip that came with the anaconda installation of Python to install those packages. Pip is just a Python program, so whichever Python interpreter is running it will be the one it uses to determine where packages should go.
Try looking at sys.path, since that gives the folders that the interpreter will look in for modules and packages.

This is one of those times where "try it!" is a good way of figuring out what is going on.

Btw, is there any particular reason why you have two Python installations? On Linux that's common, because pretty much any Linux install will install whichever version of Python the distro is using for system management scripts, but often that's not the version the user wants to run for their own programs, so they'll install a different one. But Windows doesn't use Python for its own system management, so the only way you'll have a Python installation at all is if you do it yourself.

Thanks.

No, having two Python installations is not intentional at all. When I started, I first downloaded the vanilla Python installation from the python.org. Later, I downloaded Anaconda, to try it and because it has Jupiter notebooks and all the packages already installed, and ended up in this place.

I will check sys.path and see what is there.

What would you recommend doing? Uninstalling Anaconda? Should I have not downloaded Anaconda having the regular Python installation?
 
jedishrfu said:
A lot of folks tend to use Anaconda in the Machine Learning field and its corresponding conda multi-purpose command to install packages from the Anaconda repos.

But when you're using the installed pyhtin or just not using the Anaconda distro then folks will use pip to get third party modules. However, Anaconda folks realized new folks coming to Ananconda may still want to use pip and so they provided it in their distro as well.
Thanks. I thought I was using pip that came with the regular Python installation before I downloaded Anaconda...Now I have two pips and two Python interpreters.. It sounds messy...
 
fog37 said:
What would you recommend doing?

If you only want one Python installation you should probably uninstall whichever one you don't want. Since you seem to be using the extra features in the Anaconda installation, I would say that's the one you should keep. From what you've said, it seems like all of the packages you've installed are installed with the Anaconda Python anyway, so you won't be losing anything if you just uninstall the vanilla python.org Python.
 
  • Like
Likes fog37
  • #10
Indeed, at the command line, when typing python, it is the Anaconda interpreter that gets launched, not the regular Python one. However, it talks about a conda environment not being activated and I am not there yet understanding what that does and means. I need to make sure I don't uninstall the correct Python :)

Also, I followed your advice and typed sys.path after that (see below). You mentioned that those are the folders that the interpreter look into for modules and packages. I see a lot of anaconda3 subfolders but there is one subfolder called site-packages which is located in the Python37 subfolder. Why would the Anaconda python interpreter look into a folder that pertains to the other python interpreter?

1613959760886.png
 
  • #11
fog37 said:
there is one subfolder called site-packages which is located in the Python37 subfolder.

In the Python37 subfolder of your user account. That's not the same as the system-wide site-packages for the vanilla Python from python.org that you installed. Pretty much any Python installation is going to give you some way of installing packages specific to a particular user, and those will get installed in that user's account somewhere.
 
  • #12
Thanks.

I guess it was a mistake to install both, the regular Python interpreter and Anaconda later on to try it out.
I should have removed the first Python interpreter before installing Anaconda. Anaconda has everything: the IDLE, pip, tons of modules, everything essentially.

Keeping both interpreters does not make sense.

Maybe using virtual environments would be a solutions. Those are folders containing modules, a pip, and a particular Python interpreter and they isolate their content from everything else. I can create virtual environments from either Anaconda or the regular vanilla Python...
 
  • #13
fog37 said:
Maybe using virtual environments would be a solutions.

A solution to what? You don't appear to want to have multiple different interpreters around.

In any case, virtual environments don't eliminate the confusion you appear to have been having; you still have to know which of the multiple Python interpreters you have installed is the one that the virtual environment was made from, and nothing in the virtual environment itself tells you that.

Virtual environments are typically used when different Python programs people are running (or developing) require different versions of particular modules or packages, either from each other or from the ones that come with the global Python interpreter that is installed on the system.
 
  • #14
PeterDonis said:
A solution to what? You don't appear to want to have multiple different interpreters around.

In any case, virtual environments don't eliminate the confusion you appear to have been having; you still have to know which of the multiple Python interpreters you have installed is the one that the virtual environment was made from, and nothing in the virtual environment itself tells you that.

Virtual environments are typically used when different Python programs people are running (or developing) require different versions of particular modules or packages, either from each other or from the ones that come with the global Python interpreter that is installed on the system.

Ok, I see how virtual environments are not the solution but just a way to keep specific packages/modules aside for a specific project.

I guess what I call the problem is to have both a python.exe interpreter version 3.7.3 from the Anaconda installation and another Python interpreter version 3.7.9 from the regular Python installation on the same machine.
When typing python at the command line, not matter the current directory, the anaconda python interpreter is the one launched, not the other. Both python interpreters are stored in folders that are on the Windows PATH variable...Not sure why the Anaconda Python is the one starting and not the other..

I just wonder if having the other Python version 3.7.9 interpreter could create issues with modules, etc.

Not sure how to launch the version 3.7.9 Python interpreter directly from the command line yet. I can use the 3.7.9 version if I start the IDLE from the desktop menu:
1614014532877.png


Overall, I noticed, that working inside the IDLE that uses the 3.7.9 Python interpreter, I can still import and use
the modules located in the Anaconda subfolders. Those modules were installed using the pip coming with Anaconda. I guess 3.7.9 Python interpreter can have access to all those modules too...
 
  • #15
fog37 said:
what I call the problem is to have both a python.exe interpreter version 3.7.3 from the Anaconda installation and another Python interpreter version 3.7.9 from the regular Python installation on the same machine.

Why would you want to?

Sometimes people who are doing software development (i.e., development of software they are going to ship to others) do want different Python interpreter versions on their machine so they can test software they are developing with different versions; but even then those will be different 3.x versions, such as 3.6 vs. 3.7 vs. 3.8, not different releases of the same 3.x version.

But if all you want to do is run Python programs for yourself (ones that you write or ones that others give you), I'm not sure why you would need more than one Python interpreter installed.

fog37 said:
Both python interpreters are stored in folders that are on the Windows PATH variable...Not sure why the Anaconda Python is the one starting and not the other..

I would assume it's because the folder it is in appears first on the Windows PATH. Windows checks those folders in the order given.

fog37 said:
working inside the IDLE that uses the 3.7.9 Python interpreter, I can still import and use
the modules located in the Anaconda subfolders. Those modules were installed using the pip coming with Anaconda. I guess 3.7.9 Python interpreter can have access to all those modules too...

Again, the way to check this is to look at sys.path. You should be able to do that in IDLE.
 
  • Like
Likes fog37
  • #16
PeterDonis said:
Why would you want to?

thanks.

I really did not want to do that. I have no desire to have two interpreters at all.

When I started, I just downloaded Python from python.org and got the version 3.7.9 interpreter. So far so good.
Later, I decided I wanted to try Anaconda because of all its modules, etc. so I downloaded it. And that installation apparently gave me the other interpreter, the version 3.7.3 interpreter. I don't recall if I made some wrong selection during the Anaconda installation process.

I guess a beginner like me, wanting to try the Anaconda distro, should have first uninstalled the regular Python interpreter and then downloaded Anaconda with its interpreter and everything else...
 
  • #17
fog37 said:
I guess a beginner like me, wanting to try the Anaconda distro, should have first uninstalled the regular Python interpreter and then downloaded Anaconda with its interpreter and everything else...
I don't see any reason why you can't uninstall the one you don't want now. If there is some technical reason, then I would suggest editing your PATH and PYTHONPATH to remove any directories associated with the Python that you don't want to use.

At least Windows users have the luxury of being able to have only one Python installation. Unix/Linux/MacOS users have to deal with a pre-installed Python that might not be the version they want (e.g. MacOS shipped with Python2 and not Python3 until last year) and which they might not want to modify by installing packages, for fear of screwing up whatever the OS is doing with Python.
 
  • Like
Likes fog37

Similar threads

Replies
8
Views
2K
Replies
2
Views
766
Replies
1
Views
7K
Replies
9
Views
1K
Replies
2
Views
2K
Replies
17
Views
7K
Replies
8
Views
1K
Replies
25
Views
3K
Back
Top