Skip to main content

System package manager APIs in strongly typed Python

Project description

pydantic-pkgr
System package manager APIs in strongly typed Python

GitHub GitHub Last Commit GitHub Issues

PyPI Python Version Django Version Downloads

This is a Python library for installing & managing packages with a variety of package managers.

pip install pydantic-pkgr

📦 Provides consistent cross-platform interfaces for dependency resolution & installation at runtime
🌈 Supports django >= 4.0 and django-jsonform out-of-the-box
✨ Built with pydantic v2 for strong static typing guarantees and json import/export compatibility

Built by ArchiveBox to install and auto-update our internet-archiving dependencies at runtime (like chrome, wget, curl) across macOS/Linux/Docker.


Source Code: https://github.com/ArchiveBox/pydantic-pkgr/

Documentation: https://github.com/ArchiveBox/pydantic-pkgr/blob/main/README.md


from pydantic_pkgr import AptProvider

apt = AptProvider()
curl = apt.install(bin_name='curl')        # Example: Install curl using the apt provider
print(curl.loaded_provider)                # 'apt'
print(curl.loaded_abspath)                 # Path('/usr/bin/curl')
print(curl.loaded_version)                 # SemVer('7.81.0')
curl.exec(['--version'])                   # curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 ...


from pydantic_pkgr import Binary, BinName, BinProvider

class CurlBinary(Binary):                  # Example: Define a re-usable Binary supporting multiple install methods
    name: BinName = 'curl'
    providers_supported: list[BinProvider] = [
        BrewProvider(), AptProvider(), EnvProvider(),
    ]

curl = CurlBinary().load_or_install()      # Example: Check for existing binary, install if missing
print(curl.loaded_provider)                # 'brew'
print(curl.loaded_abspath)                 # Path('/opt/homebrew/bin/curl')
print(curl.loaded_version)                 # SemVer('8.4.0')
curl.exec(['--version'])                   # curl 8.4.0 (x86_64-apple-darwin23.0) libcurl/8.4.0 ...

print(curl.model_dump_json(indent=4))      # ... everything can also be dumped/loaded as json
...

Supported Package Managers

So far it supports installing/finding installed/updating/removing packages on Linux/macOS with:

  • apt
  • brew
  • pip
  • npm
  • env (looks for existing version of binary in user's $PATH at runtime)
  • vendor (you can bundle vendored copies of packages you depend on within your source)

Planned: docker, cargo, nix, apk, go get, gem, pkg, and more using ansible/pyinfra...


Usage

pip install pydantic-pkgr

BinProvider

This type represents a "provider of binaries", e.g. a package manager like apt/pip/npm, or env (which finds binaries in your $PATH).

BinProviders implement the following interface:

  • load(bin_name: str), install(bin_name: str), load_or_install(bin_name: str) -> Binary
  • install(bin_name: str)
  • get_abspath(bin_name: str) -> Path('/absolute/path/to/bin')
  • get_version(bin_name: str) -> SemVer('1.0.0')
  • get_subdeps(bin_name: str) -> InstallStr('somepackage some-extras')
import platform
from typing import List


from pydantic_pkgr.binproviders import EnvProvider, PipProvider, AptProvider, BrewProvider


### Example: Finding an existing install of bash using the system $PATH environment
env = EnvProvider()
bash = env.load(bin_name='bash')
print(bash.loaded_abspath)            # Path('/opt/homebrew/bin/bash')
print(bash.loaded_version)            # SemVer('5.2.26')
bash.exec(['-c', 'echo hi'])          # hi

### Example: Installing curl using the apt package manager
apt = AptProvider()
curl = apt.install(bin_name='curl')
print(curl.loaded_version)            # Path('/usr/bin/curl')
print(curl.loaded_version)            # SemVer('8.4.0')
curl.exec(['--version'])              # curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 ...

### Example: Finding/Installing django with pip (w/ customized binpath resolution behavior)
pip = PipProvider(
    abspath_provider={'*': lambda bin_name, **context: inspect.getfile(bin_name)},  # use python inspect to get path instead of os.which
)
django_bin = pip.load_or_install(bin_name='django')
print(django_bin.loaded_abspath)      # Path('/usr/lib/python3.10/site-packages/django/__init__.py')
print(django_bin.loaded_version)      # SemVer('5.0.2')

Binary

This type represents a single binary dependency aka a package (e.g. wget, curl, ffmpeg, etc.).
It can define one or more BinProviders that it supports, along with overrides to customize the behavior for each.

Binarys implement the following interface:

  • load(), install(), load_or_install() -> Binary
  • loaded_provider: str
  • loaded_abspath: Path
  • loaded_version: SemVer
from pydantic_pkgr import BinProvider, Binary, BinProviderName, BinName, ProviderLookupDict, SemVer

### Example: Create a re-usable class defining a binary and its providers

class YtdlpBinary(Binary):
    name: BinName = 'ytdlp'
    description: str = 'YT-DLP (Replacement for YouTube-DL) Media Downloader'

    providers_supported: List[BinProvider] = [EnvProvider(), PipProvider(), AptProvider(), BrewProvider()]
    
    # customize installed package names for specific package managers
    provider_overrides: Dict[BinProviderName, ProviderLookupDict] = {
        'pip': {'subdeps': lambda: 'yt-dlp[default,curl-cffi]'}},
        'apt': {'subdeps': lambda: 'yt-dlp ffmpeg'}},
        'brew': {'subdeps': 'some.other.module.get_brew_subdeps'}},  # also accepts dotted import path to function
    }

ytdlp = YtdlpBinary().load_or_install()
print(ytdlp.loaded_provider)              # 'brew'
print(ytdlp.loaded_abspath)               # Path('/opt/homebrew/bin/yt-dlp')
print(ytdlp.loaded_version)               # SemVer('2024.4.9')
print(ytdlp.is_valid)                     # True
from pydantic_pkgr import BinProvider, Binary, BinProviderName, BinName, ProviderLookupDict, SemVer

#### Example: Create a binary that uses Podman if available, or Docker otherwise

class DockerBinary(Binary):
    name: BinName = 'docker'

    providers_supported: List[BinProvider] = [EnvProvider(), AptProvider()]
    
    provider_overrides: Dict[BinProviderName, ProviderLookupDict] = {
        'env': {
            # prefer podman if installed (or fall back to docker)
            'abspath': lambda: os.which('podman') or os.which('docker') or os.which('docker-ce'),
        },
        'apt': {
            # install docker OR docker-ce (varies based on CPU architecture)
            'subdeps': lambda: {
                'amd64': 'docker',
                'armv7l': 'docker-ce',
                'arm64': 'docker-ce',
            }.get(platform.machine()) or 'docker',
        },
    }

docker = DockerBinary().load_or_install()
print(docker.loaded_provider)             # 'env'
print(docker.loaded_abspath)              # '/usr/local/bin/podman'
print(docker.loaded_version)              # Ž6.0.2'
print(docker.is_valid)                    # True


# You can also pass **kwargs to override properties at runtime,
# e.g. if you want to force the abspath to be at a specific path:
custom_docker = DockerBinary(loaded_abspath='~/custom/bin/podman').load()
print(custom_docker.name)                 # 'docker'
print(custom_docker.loaded_provider)      # 'env'
print(custom_docker.loaded_abspath)       # '/Users/example/custom/bin/podman'
print(custom_docker.loaded_version)       # '5.0.2'
print(custom_docker.is_valid)             # True

SemVer

from pydantic_pkgr import SemVer

### Example: Use the SemVer type directly for parsing & verifying version strings

SemVer.parse('Google Chrome 124.0.6367.208+beta_234. 234.234.123')  # SemVer(124, 0, 6367')
SemVer.parse('2024.04.05)                                           # SemVer(2024, 4, 5)
SemVer.parse('1.9+beta')                                            # SemVer(1, 9, 0)
str(SemVer(1, 9, 0))                                                # '1.9.0'



Django Usage

The pydantic ecosystem help us get auto-generated, type-checked Django fields & forms that support BinProvider and Binary.

[!TIP] For the full experience, we recommend installing these 3 excellent packages:


Django Model Usage: Store BinProvider and Binary entries in your model fields

pip install django-pydantic-field

Fore more info see the django-pydantic-field docs...

Usage in your models.py:

from django.db import models
from django_pydantic_field import SchemaField

from pydantic_pkgr import BinProvider, EnvProvider, Binary

DEFAULT_PROVIDER = EnvProvider()

class MyModel(models.Model):
    ...

    # SchemaField supports storing a single BinProvider/Binary in a field...
    favorite_binprovider: BinProvider = SchemaField(default=DEFAULT_PROVIDER)

    # ... or inside a collection type like list[...] dict[...]
    optional_binaries: list[Binary] = SchemaField(default=[])

curl = Binary(name='curl', providers=[DEFAULT_PROVIDER]).load()

obj = MyModel(optional_binaries=[curl])
obj.save()

assert obj.favorite_binprovider == DEFAULT_PROVIDER
assert obj.optional_binaries[0].provider == DEFAULT_PROVIDER

Django Admin Usage: Show read-only list of BinProviders and Binaries in Admin UI

pip install pydantic-pkgr django-admin-data-views

For more info see the django-admin-data-views docs...

Then add this to your settings.py:

INSTALLED_APPS = [
    # ...

    'pydantic_pkgr'

    'admin_data_views'

    # ...
]

ADMIN_DATA_VIEWS = {
    "NAME": "Environment",
    "URLS": [
        {
            "route": "binproviders/",
            "view": "pydantic_pkgr.views.binproviders_list_view",
            "name": "binproviders",
            "items": {
                "route": "<str:key>/",
                "view": "pydantic_pkgr.views.binprovider_detail_view",
                "name": "binprovider",
            },
        },
        {
            "route": "binaries/",
            "view": "pydantic_pkgr.views.binaries_list_view",
            "name": "binaries",
            "items": {
                "route": "<str:key>/",
                "view": "pydantic_pkgr.views.binary_detail_view",
                "name": "binary",
            },
        },
    ],
}
Note: If you override the default site admin, you must register the views manually...

admin.py:

from django.contrib import admin

class YourSiteAdmin(admin.AdminSite): """Your customized version of admin.AdminSite""" ...
custom_admin = YourSiteAdmin() custom_admin.register(get_user_model()) ...
# Register the django-admin-data-views manually on your custom site admin from admin_data_views.admin import get_app_list, get_urls, admin_data_index_view, get_admin_data_urls
custom_admin.get_app_list = get_app_list.__get__(custom_admin, YourSiteAdmin) custom_admin.get_admin_data_urls = get_admin_data_urls.__get__(custom_admin, YourSiteAdmin) custom_admin.admin_data_index_view = admin_data_index_view.__get__(custom_admin, YourSiteAdmin) custom_admin.get_urls = get_urls(custom_admin.get_urls).__get__(custom_admin, YourSiteAdmin)

Django Admin Usage: JSONFormWidget for editing BinProvider and Binary data

Install django-jsonform to get auto-generated Forms for editing BinProvider, Binary, etc. data

pip install pydantic-pkgr django-pydantic-field django-jsonform

For more info see the django-jsonform docs...

admin.py:

from django.contrib import admin
from django_jsonform.widgets import JSONFormWidget
from django_pydantic_field.v2.fields import PydanticSchemaField

class MyModelAdmin(admin.ModelAdmin):
    formfield_overrides = {PydanticSchemaField: {"widget": JSONFormWidget}}

admin.site.register(MyModel, MyModelAdmin)



Examples

Advanced: Implement your own package manager behavior by subclassing BinProvider

from subprocess import run, PIPE

from pydantic_pkgr import BinProvider, BinProviderName, BinName, SemVer

class CargoProvider(BinProvider):
    name: BinProviderName = 'cargo'
    
    def on_setup_paths(self):
        if '~/.cargo/bin' not in sys.path:
            sys.path.append('~/.cargo/bin')

    def on_install(self, bin_name: BinName, **context):
        subdeps = self.on_get_subdeps(bin_name)
        installer_process = run(['cargo', 'install', *subdeps.split(' ')], stdout=PIPE, stderr=PIPE)
        assert installer_process.returncode == 0

    def on_get_subdeps(self, bin_name: BinName, **context) -> InstallStr:
        # optionally remap bin_names to strings passed to installer 
        # e.g. 'yt-dlp' -> 'yt-dlp ffmpeg libcffi libaac'
        return bin_name

    def on_get_abspath(self, bin_name: BinName, **context) -> Path | None:
        self.on_setup_paths()
        return Path(os.which(bin_name))

    def on_get_version(self, bin_name: BinName, **context) -> SemVer | None:
        self.on_setup_paths()
        return SemVer(run([bin_name, '--version'], stdout=PIPE).stdout.decode())

cargo = CargoProvider()
rg = cargo.install(bin_name='ripgrep')
print(rg.loaded_provider)
print(rg.loaded_version)



TODO

  • Implement initial basic support for apt, brew, and pip
  • Add preinstall and postinstall hooks for things like adding apt sources and running cleanup scripts
  • Provide editability and actions via Django Admin UI using django-pydantic-field and django-jsonform
  • Write more documentation

Other Packages We Like

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pydantic_pkgr-0.1.0.tar.gz (22.3 kB view hashes)

Uploaded Source

Built Distribution

pydantic_pkgr-0.1.0-py3-none-any.whl (23.6 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page