Python upgrade
Python upgrade
Re-installing your packages after a Python version upgrade
Background
Python puts out a (minor) version upgrade every year, timed so that it can be included in that year's release of Fedora. After that, Arch Linux, and then Manjaro, upgrade Python in their repositories as soon as they practically can.
If you have installed extra Python packages from the Manjaro repositories, they will be automatically upgraded along with your Manjaro system's Python (generally referred to as "system python").
But if you have installed Python packages from elsewhere, they will not be upgraded automatically. This page is about upgrading packages that you installed from the AUR and/or from PyPI.
Packages that you installed are in directories named 'site-packages'.
- For system python at /usr/lib/python3.9/site-packages/
- In your user-space at ~/.local/lib/python3.9/site-packages/
- If you use virtual environments, each of those has a 'site-packages' directory.
Actions
pip
Before you upgrade system Python, get some info from pip, as you won't be able to get it after the upgrade.
'pip freeze' produces a list of packages that were installed with pip, which you can use later to install them again for the new version of python.
This gets packages installed by pip in system python:
sudo pip freeze > pip_list_sudo.txt
This gets packages installed by pip in your user space:
pip freeze --user > pip_list_user.txt
And this is a template for getting them from a virtual environment:
/path/to/<venv_name>/python -m pip freeze -l > pip_list_<venv_name>.txt
After the Python upgrade, the usual advice is to do something like
sudo pip install -r pip_list_sudo.txt
but don't do that, because:
- it's best not to install pip packages in system python;
- you might prefer not to re-install all of them;
- the list from 'pip freeze' includes version numbers, and you might want to manually install afresh to get current versions.
So a good course of action would be something like:
- edit the lists and remove whatever you don't want to re-install;
- for each package you want, check if it's available from the AUR, or the Manjaro repositories, and if it is, move it to another list, for installation from the other source;
- install each pip package manually, without a version number, without sudo, with '--user':
pip install <package_name> --user
AUR
You cannot prepare for this before the upgrade.
To get a list of packages to re-install, you will use
pacman -Qoq /usr/lib/python3.9
If you run this before the upgrade, it will list many packages that pacman will soon re-install for python 3.10
When you run it after the upgrade, it will get only the packages that did not get updated to 3.10, which will be those that you installed from the AUR.
Rebuilding of AUR packages can be done automatically, with pikaur
pikaur -S $(pacman -Qoq /usr/lib/python3.9) --rebuild
or with yay
yay -S $(pacman -Qoq /usr/lib/python3.9) --answerclean All
It's possible that an AUR package has not been updated to Python 3.10 yet. If so, the above commands might fail partway through, and you'll have to rebuild the rest of the packages one or a few at a time.