livecheck

Tool for overlays to update ebuilds. Inspired by the MacPorts port subcommand of the same name and nvchecker.

Internal workings

The script searches for new versions at every SRC_URI location of the ebuild that refers to the package, using logic for GitHub, PyPI, PECL, and other hosts, or the configuration in the livecheck.json file within the same package directory. Vendor archives and signature or checksum files are skipped. The first remaining entry is always consulted, and a later entry is consulted when its URL includes both the package name and the ebuild version. The highest version found across the locations is selected, and a location that is not updated upstream does not hide a newer version published at another one. When no location reports a version, metadata.xml, HOMEPAGE, Repology, and directory listings are tried in turn. The ebuild is automatically updated if --auto-update is passed.

It is recommended to use OAuth tokens for both Github and GitLab to avoid rate limiting problems with the REST API. Use your secret storage to store github.com, bitbucket.org or gitlab.com tokens with the livecheck user. See keyring to manage tokens.

Example: storing credentials

secret-tool store --label="Password for 'livecheck' on 'bitbucket.org'" service bitbucket.org username livecheck

Installation

On Gentoo, add my overlay and install:

eselect overlay enable tatsh-overlay
emerge --sync
emerge livecheck

Command line usage

Usage: livecheck [OPTIONS] [PACKAGE_NAMES]...

Options:
  -a, --auto-update            Rename and modify ebuilds.
  -d, --debug                  Enable debug logging.
  -D, --development            Include development packages.
  --dist-github-release TEXT   GitHub release tag to upload vendor dist
                               archives under.
  --dist-github-repository TEXT
                               GitHub `owner/repo` to upload vendor dist
                               archives to as release assets.
  --dist-force-upload          Force rebuild and re-upload of vendor dist
                               archives even when present.
  -e, --exclude TEXT           Exclude package(s) from updates.
  -g, --git                    Use git and pkgdev to make changes.
  -H, --hook-dir               Run a hook directory scripts with various parameters.
  -k, --keep-old               Keep old ebuild versions.
  -p, --progress               Enable progress logging.
  --package-manager [npm|pnpm|yarn]
                               Package manager to use for Node.js packages.
  -W, --working-dir DIRECTORY  Working directory. Should be a port tree root.
  --help                       Show this message and exit.

Uploading vendor dist archives to GitHub releases

When --auto-update regenerates a vendor archive (Composer, Go modules, Maven, Node modules, or NuGet packages), the file is normally written only into DISTDIR. Pass both --dist-github-repository owner/repo and --dist-github-release tag to additionally publish the archive as an asset of that release. Per-package overrides may also be set in livecheck.json with the keys dist_github_repository and dist_github_release.

By default the asset is skipped entirely (no rebuild, no upload) if a release asset with the expected filename is already present; pass --dist-force-upload to rebuild and replace it. When the release does not exist, livecheck creates it as a draft and logs a warning instructing you to publish it from the GitHub UI so Portage can fetch the assets.

Heuristic update detection

This package can do automated lookups based on commonly used hosts. Currently:

• Bitbucket
• Davinci products
• Github archives
• Github commit hashes
• Github releases
• GitLab releases
• Hex-Rays for IDA Free
• JetBrains products
• NuGet
• PECL
• Packages from Yarn and NPM
• Perl CPAN
• PyPI
• Raphnet
• Repology
• RubyGems
• SourceHut releases / commit hashes
• SourceForge

This works as long as the version system is usable with Portage's version comparison function. For anything else, see Package configuration.

Package configuration

For packages that will not work with heuristic checking, a configuration file named livecheck.json can be placed in the directory alongside the ebuild.

Configuration keys

• branch - string- The GitHub branch name to use for commits.
• composer_packages - boolean - Download composer vendor modules.
• composer_path - path - Where is 'composer.json' located (need composer_packages).
• crates - boolean - Build a Rust dependency archive with Cargo using versions from Cargo.lock.
• crates_path - path - Source subdirectory with Cargo.toml and Cargo.lock (requires crates).
• dist_github_release - string - Per-package override for --dist-github-release.
• dist_github_repository - string - Per-package override for --dist-github-repository (owner/repo).
• dotnet_packages - boolean - Build a NuGet packages vendor archive (-nuget.tar.xz).
• dotnet_project - path - Project or solution file (.csproj / .sln) used by dotnet restore.
• maven_packages - boolean - Download Maven dependencies.
• maven_path - path - Where is 'pom.xml' located (need maven_packages).
• development - bool - Include development packages.
• gomodule_packages - boolean - Download go vendor modules.
• gomodule_path - path - Where is 'go.mod' located (need gomodule_packages).
• jetbrains_packages - boolean - Update internal ID.
• keep_old - boolean - Keep old ebuild versions.
• no_auto_update - boolean - Do not allow auto-updating of this package.
• nodejs_packages - boolean - Download nodejs node_modules.
• nodejs_path - path - Where is 'package.json' located (need nodejs_packages).
• nodejs_package_manager - string - Package manager to use for Node.js packages [npm|pnpm|yarn] (defaults to npm)
• nodejs_omit_dev - boolean - Leave the development dependencies out of the node_modules archive (need nodejs_packages). Only for ebuilds that do not run a build step.
• sha_source- string - Url to get the sha value.
• stable_version- string - Regular expression to determine if it is a stable version.
• sync_version - string - Category and ebuild with version to sync.
• transformation_function - string - Function to use to transform the version string. Currently only dotize is supported. Others are for internal use.
• type - string - Only one none, davinci, regex, directory, changelog, commit, repology or checksum.

Use the pattern to adjust the version using a regular expression:

• pattern_version - string - The pattern string.
• replace_version - string - The replacement string.

Only when type is regex, directory, or changelog:

• url - URL of the document or directory listing to scan for versions. Required.

Example for type changelog:

{
  "type": "changelog",
  "url": "https://raw.githubusercontent.com/standard/standard/refs/heads/master/CHANGELOG.md"
}

Only when type is regex:

• regex - string - The regular expression to use. Required.

Only when type is repology:

• package - string - The package to search in repology. Required.

Hook directory

The hooks directory structure is subdivided into actions, currently post and pre. Within each action directory there can be several scripts that are executed in order by name.

Arguments

• Root portage directory, e.g. /var/db/repos/gentoo.
• Category and package name, e.g. dev-lang/php.
• Previous version, e.g. 8.2.32-r2.
• New version, e.g. 8.2.33.
• SHA hash of the old version. Optional.
• SHA hash of the new version. Optional.
• Date associated with the hash. Optional.

Development use

Creating new downloads

Rust crate archives use the crates setting in a package's livecheck.json:

{
  "crates": true
}

Cargo must be installed, and upstream sources must provide Cargo.toml and Cargo.lock. Set crates_path when the Cargo project is in a source subdirectory. Dependencies are downloaded with cargo vendor --locked --versioned-dirs; Git and alternative registry dependencies are currently unsupported. A missing or outdated lock file aborts the update.

Configure the ebuild to fetch ${P}-crates.tar.xz from your dist server instead of individual CARGO_CRATE_URIS, and remove its old CRATES list. The generated archive uses cargo_home/gentoo/ under WORKDIR, as expected by cargo_src_unpack. Livecheck writes the archive to DISTDIR using the same compression and optional GitHub upload helpers as other vendor archives. Post-update hooks can upload the archive to a custom dist server.

There are 2 types of downloads: file and latest commit (currently only Git is supported) and this is evident from the first download URL of the ebuild itself.

• To download a file, a search is performed by version/tag, and optionally you can include the commit of said version, including all the results in a list so that the highest one can be selected, according to the search criteria or limit.

• To locate the last commit of an ebuild, we need the SHA of the commit and the date. This is necessary to be able to adjust the name of the ebuild using the a.b.c_pYYYYMMDD version as a scheme. If a different SHA is detected the version is updated.

Set up PYTHONPATH

As root, set the environment variable PYTHONPATH to include where the livecheck module is located. Use python -m livecheck instead of livecheck to execute commands.

With a virtualenv

Run uv sync --all-groups --all-extras to set up a virtualenv.

Fully copy /etc/portage to the root of your virtualenv. Then you must fix make.profile. Also consider making changes in repos.conf if necessary.

Example:

. .venv/bin/activate
uv pip install keyrings-alt
sudo cp -R /etc/portage "${VIRTUAL_ENV}/etc/"
sudo chown -R "${USER}:" "${VIRTUAL_ENV}/etc/portage"
ln -sf "$(readlink -f /etc/portage/make.profile)" "${VIRTUAL_ENV}/etc/portage/make.profile"