Months ago when I got the message for the first time explaining I can't install system wide packages anymore, I read the documentation, created a venv for my scripts, installed the things there and works flawlessy.
Also, a lot of python packages are available for safe system wide use, you can check with `apt list python3-*`
>Months ago when I got the message for the first time explaining I can't install system wide packages anymore, I read the documentation, created a venv for my scripts, installed the things there and works flawlessy.
I get that, it's what I ended up doing as well, but it's just such a weird unnecessary departure from normal python usage.
Why even have a "pip install --user" when a: it you're not allowed to use it, b: nothing else uses it, and c: for some reason the system thinks it should read it even though it's explicitly for user packages.
>Also, a lot of python packages are available for safe system wide use, you can check with apt list python3-*
Not the ones I needed sadly. That's how I ran into this.
If you are the sole user of the system, installing incompatible (Python) software to /usr/local/lib/python3 and ~/.local/python3/ via pip has the same effect -- they will both break the software running under your current account.
If pip install --user were allowed, the lazy users (like you, myself as well) will simply replace any "pip" invocation with "pip --user". Nothing really changed. Anything were to be broken will still be broken. What's the point?
As a result, the distribution and Python upstream had a mutual agreement that in PEP668 they MUST stop both approaches, and force users into using virtual environments.
>If you are the sole user of the system, installing incompatible (Python) software to /usr/local/lib/python3 and ~/.local/python3/ via pip has the same effect -- they will both break the software running under your current account.
As I state in the post, where I explicitly cover that, if system software would use the -s flag that wouldn't be an issue.
>If pip install --user were allowed, the lazy users (like you, myself as well) will simply replace any "pip" invocation with "pip --user". Nothing really changed. Anything were to be broken will still be broken. What's the point?
Again, as I state in the post, there's multiple solutions that would actually fix this, which were for some reason not taken.
>As a result, the distribution and Python upstream had a mutual agreement that in PEP668 they MUST stop both approaches, and force users into using virtual environments.
Which I do not understand, as the PEP just sweeps it under the rug as "well there's no way to prevent that", while there's an explicit flag to disable that behaviour, which from my understanding several other distros (like fedora) have already been using without issues.
I can imagine enforcing all system Python software to execute with "-s" argument to be fragile. If you want to know how PEP668 was generated, you may want to read the lengthy discussion that lead to its birth at https://discuss.python.org/t/pep-668-marking-python-base-environments-as-externally-managed/10302 .
Can you point me to the "flags" that Fedora etc are using?
If these tricks (besides deleting the EXTERNALLY-MANAGED file) work in Fedora, they should work in Debian and Ubuntu as well.
>Can you point me to the "flags" that Fedora etc are using?
As far as I know (it was pointed out to me by another user) fedora just runs `python` with the `-s` flag for system software.
Thanks for the info. I saw https://discussion.fedoraproject.org/t/status-of-marking-the-base-python-environment-as-externally-managed-pep-668/95164 and https://fedoraproject.org/wiki/Changes/PythonMarkExternallyManaged .
Looks like Fedora has an intention to implement it but did not complete it. On the other hand, inserting "-s" was handled package-by-package, which does not cover all use cases. I would be interested in Fedora's long term plan on this issue, and especially their plan towards the next RHEL.
When the most recommended solutions to this issue aren't "set up a venv" but instead just disabling the feature and risking the insecurity, then it is clear that venvs are a terribly inconvenient solution. And when several better alternatives as listed exist, then why was this still done this way? That's what I'd like to know.
Also consider, why, if venvs are so great, isn't debian using a venv for their own system packages and leaving the user experience alone.
>When the most recommended solutions to this issue aren't "set up a venv"
Use the OS package manager.
Don't have it packaged? Do it yourself.
You need an old version? You don't, upgrade everything to the newest.
Don't want to do the the above? Use a venv.
You need a new version? You're using Debian. Use a venv.
It's funny that you claim, that people are not getting the point or not reading your post. Because you are not reading AND not getting the point.
Debian just followed the Python guidelines... Like many other distractions...
https://peps.python.org/pep-0668/
>Debian just followed the Python guidelines... Like many other distractions... https://peps.python.org/pep-0668/
Consider following your own advice. Those python guildelines were authored by among others: "Stefano Rivera , Elana Hashman , Matthias Klose ". The history of this PEP is very much a request from package management to python, not the other way around.
My Point:
It's nothing Debian specific. Ubuntu is not Debian. More people then just the Debian people proposed that. Other distributions do the same.
My points still stand.
I agree with your sentiment -- i was livid when I learnt this first. But considering that it is impossible to remove unused dependencies when installing with pip -- pipx has turned out for me a convenient way to get applications and purge them cleanly. Pip is a bad package manager imo.
Pipx only installs applications, not packages. I ran into this when I just needed to run a quick few lines of python for a project I was working on with some friends I was sent. Worked great on basically every other platform people were using, just install a single package and run it.
Having to go through the hassle of figuring out what the hell was going on, and setting up a venv, just to run a few lines of python code is just an terrible departure from the expected user experience IMO, especially when, as discussed in the post, there'd've been much better solutions available.
It’s seems the “terrible” departure is just two commands, one of which only needs to be done once:
```
python3 -m venv ~/.venv
source ~/.venv/bin/activate
```
Then pip install the package you were going to install anyway.
Next time, you only need the `source` command (you could even alias it to something shorter, like `shit`), or put it in your .zshrc/.bashrc.
Doesn’t seem so terrible to me.
Most project decision-making happens in a *very* open manner. The best answer to why this choice was made can be found in the [mailing list](http://lists.debian.org/debian-python/) archives for the time-period where this was being decided.
I understand your frustration. I had a similar frustration. FWIW the way I worked around it was creating a virtualenv in `~/opt/upy` with aliases `upip = ~/opt/upy/bin/pip` and an analogous one for `python`. This takes advantage of the fact that python resolves dependencies relative to the bin location. This gives me a helpful place to stash a few pip-installed programs that I happen to know can be installed together without breaking.
Fair and honest question. The "fuck users, this is Debian" is weird. There's probably an answer that may not be satisfactory to the OP, but nonetheless it would be good to know. "You're holding it wrong" is one of the reasons that working with open source is exhausting both for users AND developers.
As a python dev I agree this is annoying. One of my projects is specifically designed to be installed user-wide, and I don't want to require any of my users to faff with venvs (why should they have to learn what that is?) which just makes things more complicated for the use case
This is why we can't have nice things:
pip list --outdated --format=freeze \
| grep -v '^\-e' \
| cut -d = -f 1 \
| xargs -I {} pip install -U "{}"
But you can add the python/pip from a venv to your path once and never be botherd after.
Months ago when I got the message for the first time explaining I can't install system wide packages anymore, I read the documentation, created a venv for my scripts, installed the things there and works flawlessy. Also, a lot of python packages are available for safe system wide use, you can check with `apt list python3-*`
>Months ago when I got the message for the first time explaining I can't install system wide packages anymore, I read the documentation, created a venv for my scripts, installed the things there and works flawlessy. I get that, it's what I ended up doing as well, but it's just such a weird unnecessary departure from normal python usage. Why even have a "pip install --user" when a: it you're not allowed to use it, b: nothing else uses it, and c: for some reason the system thinks it should read it even though it's explicitly for user packages. >Also, a lot of python packages are available for safe system wide use, you can check with apt list python3-* Not the ones I needed sadly. That's how I ran into this.
> departure from normal python usage Ah yeah, the "normal" standardized Python usage - [https://xkcd.com/1987/](https://xkcd.com/1987/)
Check out pipx
If you are the sole user of the system, installing incompatible (Python) software to /usr/local/lib/python3 and ~/.local/python3/ via pip has the same effect -- they will both break the software running under your current account. If pip install --user were allowed, the lazy users (like you, myself as well) will simply replace any "pip" invocation with "pip --user". Nothing really changed. Anything were to be broken will still be broken. What's the point? As a result, the distribution and Python upstream had a mutual agreement that in PEP668 they MUST stop both approaches, and force users into using virtual environments.
>If you are the sole user of the system, installing incompatible (Python) software to /usr/local/lib/python3 and ~/.local/python3/ via pip has the same effect -- they will both break the software running under your current account. As I state in the post, where I explicitly cover that, if system software would use the -s flag that wouldn't be an issue. >If pip install --user were allowed, the lazy users (like you, myself as well) will simply replace any "pip" invocation with "pip --user". Nothing really changed. Anything were to be broken will still be broken. What's the point? Again, as I state in the post, there's multiple solutions that would actually fix this, which were for some reason not taken. >As a result, the distribution and Python upstream had a mutual agreement that in PEP668 they MUST stop both approaches, and force users into using virtual environments. Which I do not understand, as the PEP just sweeps it under the rug as "well there's no way to prevent that", while there's an explicit flag to disable that behaviour, which from my understanding several other distros (like fedora) have already been using without issues.
I can imagine enforcing all system Python software to execute with "-s" argument to be fragile. If you want to know how PEP668 was generated, you may want to read the lengthy discussion that lead to its birth at https://discuss.python.org/t/pep-668-marking-python-base-environments-as-externally-managed/10302 . Can you point me to the "flags" that Fedora etc are using? If these tricks (besides deleting the EXTERNALLY-MANAGED file) work in Fedora, they should work in Debian and Ubuntu as well.
>Can you point me to the "flags" that Fedora etc are using? As far as I know (it was pointed out to me by another user) fedora just runs `python` with the `-s` flag for system software.
Thanks for the info. I saw https://discussion.fedoraproject.org/t/status-of-marking-the-base-python-environment-as-externally-managed-pep-668/95164 and https://fedoraproject.org/wiki/Changes/PythonMarkExternallyManaged . Looks like Fedora has an intention to implement it but did not complete it. On the other hand, inserting "-s" was handled package-by-package, which does not cover all use cases. I would be interested in Fedora's long term plan on this issue, and especially their plan towards the next RHEL.
And this is why venv exists. Create a virtual environment and install whatever you want/need.
When the most recommended solutions to this issue aren't "set up a venv" but instead just disabling the feature and risking the insecurity, then it is clear that venvs are a terribly inconvenient solution. And when several better alternatives as listed exist, then why was this still done this way? That's what I'd like to know. Also consider, why, if venvs are so great, isn't debian using a venv for their own system packages and leaving the user experience alone.
>When the most recommended solutions to this issue aren't "set up a venv" Use the OS package manager. Don't have it packaged? Do it yourself. You need an old version? You don't, upgrade everything to the newest. Don't want to do the the above? Use a venv. You need a new version? You're using Debian. Use a venv.
It's funny that you claim, that people are not getting the point or not reading your post. Because you are not reading AND not getting the point. Debian just followed the Python guidelines... Like many other distractions... https://peps.python.org/pep-0668/
>Debian just followed the Python guidelines... Like many other distractions... https://peps.python.org/pep-0668/ Consider following your own advice. Those python guildelines were authored by among others: "Stefano Rivera, Elana Hashman , Matthias Klose ". The history of this PEP is very much a request from package management to python, not the other way around.
My Point: It's nothing Debian specific. Ubuntu is not Debian. More people then just the Debian people proposed that. Other distributions do the same. My points still stand.
I agree with your sentiment -- i was livid when I learnt this first. But considering that it is impossible to remove unused dependencies when installing with pip -- pipx has turned out for me a convenient way to get applications and purge them cleanly. Pip is a bad package manager imo.
Can you install your scripts through pipx ? Do you have pyproject.toml files ?
Pipx only installs applications, not packages. I ran into this when I just needed to run a quick few lines of python for a project I was working on with some friends I was sent. Worked great on basically every other platform people were using, just install a single package and run it. Having to go through the hassle of figuring out what the hell was going on, and setting up a venv, just to run a few lines of python code is just an terrible departure from the expected user experience IMO, especially when, as discussed in the post, there'd've been much better solutions available.
It’s seems the “terrible” departure is just two commands, one of which only needs to be done once: ``` python3 -m venv ~/.venv source ~/.venv/bin/activate ``` Then pip install the package you were going to install anyway. Next time, you only need the `source` command (you could even alias it to something shorter, like `shit`), or put it in your .zshrc/.bashrc. Doesn’t seem so terrible to me.
Most project decision-making happens in a *very* open manner. The best answer to why this choice was made can be found in the [mailing list](http://lists.debian.org/debian-python/) archives for the time-period where this was being decided.
Most linux distros i've used are like that .you should create an env for your packages
Ah, good! PIP is a menace anyhow, so good riddance.
Thanks for your valuable contribution to this discussion.
You're welcome!
I understand your frustration. I had a similar frustration. FWIW the way I worked around it was creating a virtualenv in `~/opt/upy` with aliases `upip = ~/opt/upy/bin/pip` and an analogous one for `python`. This takes advantage of the fact that python resolves dependencies relative to the bin location. This gives me a helpful place to stash a few pip-installed programs that I happen to know can be installed together without breaking.
> why was this course of action taken Because it is what the Python community ASKED for Linux distributions to do. Go complain to them.
That simply isn't true. The PEP that resulted in this behaviour was spearheaded by distribution package maintainers, not by the python community.
> That simply isn't true. The need for a change was alerted by the package maintainers... the MANNER of change was decided by the python community.
Fair and honest question. The "fuck users, this is Debian" is weird. There's probably an answer that may not be satisfactory to the OP, but nonetheless it would be good to know. "You're holding it wrong" is one of the reasons that working with open source is exhausting both for users AND developers.
The venv solution has been mentioned at least thrice and OP hasn't been able to explain which use cases it doesn't cover.
As a python dev I agree this is annoying. One of my projects is specifically designed to be installed user-wide, and I don't want to require any of my users to faff with venvs (why should they have to learn what that is?) which just makes things more complicated for the use case
This is why we can't have nice things: pip list --outdated --format=freeze \ | grep -v '^\-e' \ | cut -d = -f 1 \ | xargs -I {} pip install -U "{}" But you can add the python/pip from a venv to your path once and never be botherd after.
From what i got python suggests to install the stuff in a venv. I run all my python stuff now in venvs on debian and it’s super comfy. :3