Automated, robust apt-get mirror selection for Debian and Ubuntu
The apt-smart package automates robust apt-get mirror (a.k.a Repositories, Sources) selection for Debian , Ubuntu and Linux Mint by enabling smart discovery of available mirrors, smart ranking of available mirrors, automatic switching between mirrors and robust package list updating (see features). It’s currently tested on Python 2.7, 3.4, 3.5, 3.6, 3.7, 3.8 and PyPy (although test coverage is still rather low, see status).
As a successor of apt-mirror-updater, apt-smart has many improvements in intelligence, speed, accuracy and robustness (see changelog) when offering the best mirror for you. It has a plan to optionally be a set-and-forget smart daemon: running in the background as a reverse proxy always redirecting to the best mirror without root privilege. It also has a plan to support other distros like: Linux Mint (Done!) , ROS…
- Smart discovery of available mirrors
- Debian , Ubuntu and Linux Mint mirrors are discovered automatically by querying the Debian mirror list or the Ubuntu mirror list1 or the Ubuntu mirror list2 or the Linux Mint mirror list (the applicable mirror list is automatically selected based on the current platform). It can smartly get mirrors within the country which the user is in.
- Smart ranking of available mirrors
- Discovered mirrors are ranked by bandwidth (to pick the fastest mirror) and whether they’re up-to-date and excluded if they’re being updated (see issues with mirror updates). e.g. with –list-mirrors flag it would output like this:
----------------------------------------------------------------------------------------------------- | Rank | Mirror URL | Available? | Updating? | Last updated | Bandwidth | ----------------------------------------------------------------------------------------------------- | 1 | http://archive.ubuntu.com/ubuntu | Yes | No | Up to date | 16.95 KB/s | | 2 | http://mirrors.cqu.edu.cn/ubuntu | Yes | No | 3 hours behind | 427.43 KB/s | | 3 | http://mirrors.nju.edu.cn/ubuntu | Yes | No | 5 hours behind | 643.27 KB/s | | 4 | http://mirrors.tuna.tsinghua.e...| Yes | No | 5 hours behind | 440.09 KB/s | | 5 | http://mirrors.cn99.com/ubuntu | Yes | No | 13 hours behind | 2.64 MB/s | | 6 | http://mirrors.huaweicloud.com...| Yes | No | 13 hours behind | 532.01 KB/s | | 7 | http://mirrors.dgut.edu.cn/ubuntu| Yes | No | 13 hours behind | 328.25 KB/s | | 8 | http://mirrors.aliyun.com/ubuntu | Yes | No | 23 hours behind | 1.06 MB/s | | 9 | http://ftp.sjtu.edu.cn/ubuntu | Yes | No | 23 hours behind | 647.2 KB/s | | 10 | http://mirrors.yun-idc.com/ubuntu| Yes | No | 23 hours behind | 526.6 KB/s | | 11 | http://mirror.lzu.edu.cn/ubuntu | Yes | No | 23 hours behind | 210.99 KB/s | | 12 | http://mirrors.ustc.edu.cn/ubuntu| Yes | Yes | 8 hours behind | 455.02 KB/s | | 13 | http://mirrors.sohu.com/ubuntu | No | No | Unknown | 90.28 bytes/s | -----------------------------------------------------------------------------------------------------
- Automatic switching between mirrors
- The main mirror configured in /etc/apt/sources.list can be changed with a single command. The new (to be configured) mirror can be selected automatically or configured explicitly by the user.
- Robust package list updating
- Several apt-get subcommands can fail if the current mirror is being updated (see issues with mirror updates) and apt-smart tries to work around this by wrapping apt-get update to retry on failures and automatically switch to a different mirror when it looks like the current mirror is being updated (because I’ve seen such updates take more than 15 minutes and it’s not always acceptable to wait for so long, especially in automated solutions).
On the one hand the apt-smart package was developed based on quite a few years of experience in using apt-get on Debian and Ubuntu systems. On the other hand the Python package itself is relatively new: it was developed and published in Sep 2019. As such:
Until apt-smart has been rigorously tested I consider it a proof of concept (beta software) so if it corrupts your system you can’t complain that you weren’t warned! The worst that can happen (assuming you trust my judgement ;-) is that /etc/apt/sources.list is corrupted however a backup copy is made before any changes are applied, so I don’t see how this can result in irreversible corruption.
I’m working on an automated test suite but at the moment I’m still a bit fuzzy on how to create representative tests for the error handling code paths (also, writing a decent test suite requires a significant chunk of time :-).
The apt-smart package is available on PyPI which means installation should be as simple as (paste all below commands together into terminal):
sudo apt update sudo apt install python-pip python-setuptools python-wheel -y # install python-pip and so on without asking pip install --user apt-smart # --user flag means install to per user site-packages directory(see below) echo "export PATH=\$(python -c 'import site; print(site.USER_BASE + \"/bin\")'):\$PATH" >> ~/.bashrc source ~/.bashrc # set per user site-packages directory to PATH
There’s actually a multitude of ways to install Python packages (e.g. the per user site-packages directory, virtual environments or just installing system wide) and I have no intention of getting into that discussion here, so if this intimidates you then read up on your options before returning to these instructions ;-).
If a new version of apt-smart has been released, you can upgrade it via:
pip install --user apt-smart --upgrade
Note. apt-smart is a helper for the apt tool. It is NOT a replacement for apt (or for apt-get). So, apt-smart should not be run instead of either of those commands. Nor should apt-smart be run with sudo or via su; if apt-smart happens to need root privilege in order for it to continue (in order that it may, for example, change sources.list), then it will prompt for a password.
There are two ways to use the apt-smart package: As the command line program apt-smart and as a Python API. For details about the Python API please refer to the API documentation available on Read the Docs. The command line interface is described below.
Usage: apt-smart [OPTIONS]
The apt-smart program automates robust apt-get mirror selection for Debian and Ubuntu by enabling discovery of available mirrors, ranking of available mirrors, automatic switching between mirrors and robust package list updating.
|-r, --remote-host=SSH_ALIAS||Operate on a remote system instead of the local system. The SSH_ALIAS argument gives the SSH alias of the remote host. It is assumed that the remote account has root privileges or password-less sudo access.|
|-f, --find-current-mirror||Determine the main mirror that is currently configured in /etc/apt/sources.list and report its URL on standard output.|
|-F, --file-to-read=local_file_absolute_path||Read a local absolute path (path and filename must NOT contain whitespace) file containing custom mirror URLs (one URL per line) to add custom mirrors to rank.|
|-b, --find-best-mirror||Discover available mirrors, rank them, select the best one and report its URL on standard output.|
|-l, --list-mirrors||List available (ranked) mirrors on the terminal in a human readable format.|
|-L, --url-char-len=int||An integer to specify the length of chars in mirrors’ URL to display when using --list-mirrors, default is 34|
|-c, --change-mirror=MIRROR_URL||Update /etc/apt/sources.list to use the given MIRROR_URL.|
|-a, --auto-change-mirror||Discover available mirrors, rank the mirrors by connection speed and update status and update /etc/apt/sources.list to use the best available mirror.|
|-u, --update, --update-package-lists||Update the package lists using “apt-get update”, retrying on failure and automatically switch to a different mirror when it looks like the current mirror is being updated.|
|-U, --ubuntu||Ubuntu mode for Linux Mint to deal with upstream Ubuntu mirror instead of Linux Mint mirror. e.g. --auto-change-mirror --ubuntu will auto-change Linux Mint’s upstream Ubuntu mirror|
|-x, --exclude=PATTERN||Add a pattern to the mirror selection blacklist. PATTERN is expected to be a shell pattern (containing wild cards like “?” and “*”) that is matched against the full URL of each mirror.|
|-v, --verbose||Increase logging verbosity (can be repeated).|
|-V, --version||Show version number and Python version.|
|-R, --create-chroot=local_dir_absolute_path||Create chroot with the best mirror in a local directory with absolute_path|
|-q, --quiet||Decrease logging verbosity (can be repeated).|
Show this message and exit.
The most frequent failure that we run into is apt-get update crapping out with ‘hash sum mismatch’ errors (see also Debian bug #624122). When this happens a file called Archive-Update-in-Progress-* can sometimes be found on the index page of the mirror that is being used (see also Debian bug #110837). I’ve seen these situations last for more than 15 minutes.
My working theory about these ‘hash sum mismatch’ errors is that they are caused by the fact that mirror updates aren’t atomic, apparently causing apt-get update to download a package list whose datafiles aren’t consistent with each other. If this assumption proves to be correct (and also assuming that different mirrors are updated at different times :-) then the command apt-smart --update-package-lists should work around this annoying failure mode (by automatically switching to a different mirror when ‘hash sum mismatch’ errors are encountered).
Publishing apt-smart to the world is my attempt to contribute to this situation instead of complaining in bug trackers (see above) where no robust and automated solution is emerging (at the time of writing). Who knows, maybe some day these issues will be resolved by moving logic similar to what I’ve implemented here into apt-get itself. Of course it would also help if mirror updates were atomic…
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size apt_smart-7.1.3-py2.py3-none-any.whl (51.2 kB)||File type Wheel||Python version py2.py3||Upload date||Hashes View|
|Filename, size apt-smart-7.1.3.tar.gz (59.2 kB)||File type Source||Python version None||Upload date||Hashes View|
Hashes for apt_smart-7.1.3-py2.py3-none-any.whl