Python upgrade

Python upgrade

Re-installing your packages after a Python version upgrade

Background

Python puts out a (minor) version upgrade every year, timed so that it can be included in that year's release of Fedora. After that, Arch Linux, and then Manjaro, upgrade Python in their repositories as soon as they practically can.

If you have installed extra Python packages from the Manjaro repositories, they will be automatically upgraded along with your Manjaro system's Python (generally referred to as "system python").

But if you have installed Python packages from elsewhere, they will not be upgraded automatically. This page is about upgrading packages that you installed from the AUR and/or from PyPI.

Packages that you installed are in directories named 'site-packages'.

• For system python at /usr/lib/python3.9/site-packages/
• In your user-space at ~/.local/lib/python3.9/site-packages/
• If you use virtual environments, each of those has a 'site-packages' directory.

Actions

pip

Before you upgrade system Python, get some info from pip, as you won't be able to get it after the upgrade.

'pip freeze' produces a list of packages that were installed with pip, which you can use later to install them again for the new version of python.

This gets packages installed by pip in system python:

sudo pip freeze > pip_list_sudo.txt

This gets packages installed by pip in your user space:

pip freeze --user > pip_list_user.txt

And this is a template for getting them from a virtual environment:

/path/to/<venv_name>/python -m pip freeze -l > pip_list_<venv_name>.txt

After the Python upgrade, the usual advice is to do something like

sudo pip install -r pip_list_sudo.txt

but don't do that, because:

• it's best not to install pip packages in system python;
• you might prefer not to re-install all of them;
• the list from 'pip freeze' includes version numbers, and you might want to manually install afresh to get current versions.

So a good course of action would be something like:

• edit the lists and remove whatever you don't want to re-install;
• for each package you want, check if it's available from the AUR, or the Manjaro repositories, and if it is, move it to another list, for installation from the other source;
• install each pip package manually, without a version number, without sudo, with '--user':

pip install <package_name> --user

AUR

You cannot prepare for this before the upgrade.

To get a list of packages to re-install, you will use

pacman -Qoq /usr/lib/python3.9

If you run this before the upgrade, it will list many packages that pacman will soon re-install for python 3.10

When you run it after the upgrade, it will get only the packages that did not get updated to 3.10, which will be those that you installed from the AUR.

Rebuilding of AUR packages can be done automatically, with pikaur

pikaur -S $(pacman -Qoq /usr/lib/python3.9) --rebuild

or with yay

yay -S $(pacman -Qoq /usr/lib/python3.9) --answerclean All

It's possible that an AUR package has not been updated to Python 3.10 yet. If so, the above commands might fail partway through, and you'll have to rebuild the rest of the packages one or a few at a time.

See Also

pip commands