System package manager APIs in strongly typed Python
Project description
pydantic-pkgr 📦 apt brew pip npm ₊₊₊
Simple Pydantic interfaces for package managers + installed binaries.
This is a Python library for installing & managing packages locally with a variety of package managers.
It's designed to help Python apps check for and install local binaries at runtime.
pip install pydantic-pkgr
✨ Built with
pydanticv2 for strong static typing guarantees and json import/export compatibility
📦 Provides consistent cross-platform interfaces for dependency resolution & installation at runtime
🌈 Supportsdjango>= 4.0,django-ninja, and OpenAPI +django-jsonformout-of-the-box
🦄 Usespyinfra/ansiblefor the actual install operations whenever possible (with internal fallbacks)
Built by ArchiveBox to install & auto-update our extractor dependencies at runtime (chrome, wget, curl, etc.) on macOS/Linux/Docker.
[!WARNING] This is [ALPHA] software under active development, not ready for production use yet.
Source Code: https://github.com/ArchiveBox/pydantic-pkgr/
Documentation: https://github.com/ArchiveBox/pydantic-pkgr/blob/main/README.md
from pydantic_pkgr import AptProvider, Binary
apt = AptProvider()
curl = apt.install('curl')
print(curl.abspath, curl.version, curl.provider, curl.is_valid) # Path('/usr/bin/curl') SemVer('7.81.0') 'apt' True
proc = curl.exec(['-fsSL', 'https://example.com']) # <!doctype html>...
from pydantic_pkgr import Binary, BinProvider, BrewProvider, EnvProvider
# Example: Create a re-usable curl Binary object that defines its install methods
curl = Binary(name='curl', providers=[BrewProvider(), EnvProvider()])
# or for nicer type checking ergonomics, use class-based definitions:
class CurlBinary(Binary):
name: str = 'curl'
providers list[BinProvider] = [BrewProvider(), EnvProvider()]
curl = CurlBinary()
# it works the same either way
curl = curl.install()
print(curl.abspath, curl.version, curl.provider, curl.is_valid) # Path('/opt/homebrew/bin/curl') SemVer('8.4.0') 'brew' True
curl.exec(['--version']) # curl 8.4.0 (x86_64-apple-darwin23.0) libcurl/8.4.0 ...
from pyinfra.operations import apt
from pydantic_pkgr import Binary
# Example: Verify & use packages installed by other tools (e.g. pyinfra/ansible)
apt.packages(name="Install ffmpeg", packages=['ffmpeg'], _sudo=True)
# Load it as a Binary provides a nice type-checkable, validated, serializable handle for it
ffmpeg = Binary(name='ffmpeg').load()
ffmpeg.exec(['-i', 'input.mp4', 'output.avi'])
print(ffmpeg) # name=ffmpeg abspath=/usr/bin/ffmpeg version=3.3.0 is_valid=True
print(ffmpeg.model_dump_json()) # ... everything can also be dumped/loaded as json
print(ffmpeg.model_json_schema()) # ... all types provide OpenAPI-ready JSON schemas
Supported Package Managers
So far it supports installing/finding installed/updating/removingLinux/macOS with:
apt(Ubuntu/Debian/etc.)brew(macOS/Linux)pip(Linux/macOS/Windows)npm(Linux/macOS/Windows)env(looks for existing version of binary in user's$PATHat runtime)vendor(you can bundle vendored copies of packages you depend on within your source)
Planned: docker, cargo, nix, apk, go get, gem, pkg, and more using ansible/pyinfra...
Usage
pip install pydantic-pkgr
BinProvider
Implementations: EnvProvider, AptProvider, BrewProvider, PipProvider, NpmProvider
This type represents a "provider of binaries", e.g. a package manager like apt/pip/npm, or env (which finds binaries in your $PATH).
BinProviders implement the following interface:
load(bin_name: str),install(bin_name: str),load_or_install(bin_name: str)->Binaryinstall(bin_name: str)get_abspath(bin_name: str) -> Path('/absolute/path/to/bin')get_version(bin_name: str) -> SemVer('1.0.0')get_subdeps(bin_name: str) -> InstallStr('somepackage some-extras')
import platform
from typing import List
from pydantic_pkgr import EnvProvider, PipProvider, AptProvider, BrewProvider
### Example: Finding an existing install of bash using the system $PATH environment
env = EnvProvider()
bash = env.load(bin_name='bash')
print(bash.abspath) # Path('/opt/homebrew/bin/bash')
print(bash.version) # SemVer('5.2.26')
bash.exec(['-c', 'echo hi']) # hi
### Example: Installing curl using the apt package manager
apt = AptProvider()
curl = apt.install(bin_name='curl')
print(curl.version) # Path('/usr/bin/curl')
print(curl.version) # SemVer('8.4.0')
curl.exec(['--version']) # curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 ...
### Example: Finding/Installing django with pip (w/ customized binpath resolution behavior)
pip = PipProvider(
abspath_provider={'*': lambda bin_name, **context: inspect.getfile(bin_name)}, # use python inspect to get path instead of os.which
)
django_bin = pip.load_or_install(bin_name='django')
print(django_bin.abspath) # Path('/usr/lib/python3.10/site-packages/django/__init__.py')
print(django_bin.version) # SemVer('5.0.2')
Binary
This type represents a single binary dependency aka a package (e.g. wget, curl, ffmpeg, etc.).
It can define one or more BinProviders that it supports, along with overrides to customize the behavior for each.
Binarys implement the following interface:
load(),install(),load_or_install()->Binaryprovider: BinProviderName(BinProviderName == str)abspath: Pathversion: SemVer
from pydantic_pkgr import BinProvider, Binary, BinProviderName, BinName, ProviderLookupDict, SemVer
### Example: Create a re-usable class defining a binary and its providers
class YtdlpBinary(Binary):
name: BinName = 'ytdlp'
description: str = 'YT-DLP (Replacement for YouTube-DL) Media Downloader'
providers_supported: List[BinProvider] = [EnvProvider(), PipProvider(), AptProvider(), BrewProvider()]
# customize installed package names for specific package managers
provider_overrides: Dict[BinProviderName, ProviderLookupDict] = {
'pip': {'subdeps': lambda: 'yt-dlp[default,curl-cffi]'}},
'apt': {'subdeps': lambda: 'yt-dlp ffmpeg'}},
'brew': {'subdeps': 'some.other.module.get_brew_subdeps'}}, # also accepts dotted import path to function
}
ytdlp = YtdlpBinary().load_or_install()
print(ytdlp.provider) # 'brew'
print(ytdlp.abspath) # Path('/opt/homebrew/bin/yt-dlp')
print(ytdlp.version) # SemVer('2024.4.9')
print(ytdlp.is_valid) # True
from pydantic_pkgr import BinProvider, Binary, BinProviderName, BinName, ProviderLookupDict, SemVer
#### Example: Create a binary that uses Podman if available, or Docker otherwise
class DockerBinary(Binary):
name: BinName = 'docker'
providers_supported: List[BinProvider] = [EnvProvider(), AptProvider()]
provider_overrides: Dict[BinProviderName, ProviderLookupDict] = {
'env': {
# example: prefer podman if installed (falling back to docker)
'abspath': lambda: os.which('podman') or os.which('docker') or os.which('docker-ce'),
},
'apt': {
# example: vary installed package name based on your CPU architecture
'subdeps': lambda: {
'amd64': 'docker',
'armv7l': 'docker-ce',
'arm64': 'docker-ce',
}.get(platform.machine(), 'docker'),
},
}
docker = DockerBinary().load_or_install()
print(docker.provider) # 'env'
print(docker.abspath) # Path('/usr/local/bin/podman')
print(docker.version) # SemVer('6.0.2')
print(docker.is_valid) # True
# You can also pass **kwargs to override properties at runtime,
# e.g. if you want to force the abspath to be at a specific path:
custom_docker = DockerBinary(abspath='~/custom/bin/podman').load()
print(custom_docker.name) # 'docker'
print(custom_docker.provider) # 'env'
print(custom_docker.abspath) # Path('/Users/example/custom/bin/podman')
print(custom_docker.version) # SemVer('5.0.2')
print(custom_docker.is_valid) # True
SemVer
from pydantic_pkgr import SemVer
### Example: Use the SemVer type directly for parsing & verifying version strings
SemVer.parse('Google Chrome 124.0.6367.208+beta_234. 234.234.123') # SemVer(124, 0, 6367')
SemVer.parse('2024.04.05) # SemVer(2024, 4, 5)
SemVer.parse('1.9+beta') # SemVer(1, 9, 0)
str(SemVer(1, 9, 0)) # '1.9.0'
These types are all meant to be used library-style to make writing your own apps easier.
e.g. you can use it to build things like:playwright install --with-deps)
Django Usage
The pydantic ecosystem help us get auto-generated, type-checked Django fields & forms
that support BinProvider and Binary.
[!TIP] For the full experience, we recommend installing these 3 excellent packages:
django-admin-data-viewsdjango-pydantic-fielddjango-jsonform
pip install pydantic-pkgr django-admin-data-views django-pydantic-field django-jsonform
Django Model Usage: Store BinProvider and Binary entries in your model fields
pip install django-pydantic-field
Fore more info see the django-pydantic-field docs...
Usage in your models.py:
from django.db import models
from django_pydantic_field import SchemaField
from pydantic_pkgr import BinProvider, EnvProvider, Binary, SemVer
DEFAULT_PROVIDER = EnvProvider()
class Dependency(models.Model):
"""Example model for storing information about a dependency"""
name = models.CharField(max_length=63)
min_version: SemVer = SchemaField(default=(0,0,1))
binaries: list[Binary] = SchemaField(default=[])
default_binprovider: BinProvider = SchemaField(default=DEFAULT_PROVIDER)
curl = Binary(name='curl', providers=[DEFAULT_PROVIDER]).load()
obj = Dependency(default_binprovider=DEFAULT_PROVIDER, binaries=[curl], min_version=SemVer('0.1.5'))
obj.save()
assert obj.default_binprovider == DEFAULT_PROVIDER
assert obj.binaries[0].provider == DEFAULT_PROVIDER
For a full example see our provided django_example_project/...
Django Admin Usage: Show read-only list of Binaries in Admin UI
pip install pydantic-pkgr django-admin-data-views
For more info see the django-admin-data-views docs...
Then add this to your settings.py:
INSTALLED_APPS = [
# ...
'admin_data_views'
'pydantic_pkgr'
# ...
]
# point these to a function that gets the list of all binaries / a single binary
PYDANTIC_PKGR_GET_ALL_BINARIES = 'pydantic_pkgr.views.get_all_binaries'
PYDANTIC_PKGR_GET_BINARY = 'pydantic_pkgr.views.get_binary'
ADMIN_DATA_VIEWS = {
"NAME": "Environment",
"URLS": [
{
"route": "binaries/",
"view": "pydantic_pkgr.views.binaries_list_view",
"name": "binaries",
"items": {
"route": "<str:key>/",
"view": "pydantic_pkgr.views.binary_detail_view",
"name": "binary",
},
},
# Coming soon: binprovider_list_view + binprovider_detail_view ...
],
}
For a full example see our provided django_example_project/...
Note: If you override the default site admin, you must register the views manually...
admin.py:
class YourSiteAdmin(admin.AdminSite):
"""Your customized version of admin.AdminSite"""
...
custom_admin = YourSiteAdmin()
custom_admin.register(get_user_model())
...
from pydantic_pkgr.admin import register_admin_views
register_admin_views(custom_admin)
Django Admin Usage: JSONFormWidget for editing BinProvider and Binary data
BinProvider and Binary data
[!IMPORTANT] This feature is coming soon but is blocked on a few issues being fixed first:
Install django-jsonform to get auto-generated Forms for editing BinProvider, Binary, etc. data
pip install django-pydantic-field django-jsonform
For more info see the django-jsonform docs...
admin.py:
from django.contrib import admin
from django_jsonform.widgets import JSONFormWidget
from django_pydantic_field.v2.fields import PydanticSchemaField
class MyModelAdmin(admin.ModelAdmin):
formfield_overrides = {PydanticSchemaField: {"widget": JSONFormWidget}}
admin.site.register(MyModel, MyModelAdmin)
For a full example see our provided django_example_project/...
Examples
Advanced: Implement your own package manager behavior by subclassing BinProvider
from subprocess import run, PIPE
from pydantic_pkgr import BinProvider, BinProviderName, BinName, SemVer
class CargoProvider(BinProvider):
name: BinProviderName = 'cargo'
def on_setup_paths(self):
if '~/.cargo/bin' not in sys.path:
sys.path.append('~/.cargo/bin')
def on_install(self, bin_name: BinName, **context):
subdeps = self.on_get_subdeps(bin_name)
installer_process = run(['cargo', 'install', *subdeps.split(' ')], stdout=PIPE, stderr=PIPE)
assert installer_process.returncode == 0
def on_get_subdeps(self, bin_name: BinName, **context) -> InstallStr:
# optionally remap bin_names to strings passed to installer
# e.g. 'yt-dlp' -> 'yt-dlp ffmpeg libcffi libaac'
return bin_name
def on_get_abspath(self, bin_name: BinName, **context) -> Path | None:
self.on_setup_paths()
return Path(os.which(bin_name))
def on_get_version(self, bin_name: BinName, **context) -> SemVer | None:
self.on_setup_paths()
return SemVer(run([bin_name, '--version'], stdout=PIPE).stdout.decode())
cargo = CargoProvider()
rg = cargo.install(bin_name='ripgrep')
print(rg.provider) # 'cargo'
print(rg.version) # SemVer(14, 1, 0)
TODO
- Implement initial basic support for
apt,brew, andpip - Provide editability and actions via Django Admin UI using
django-pydantic-fieldanddjango-jsonform - Implement
updateandremoveactions on BinProviders - Add
preinstallandpostinstallhooks for things like addingaptsources and running cleanup scripts - Implement more package managers
Other Packages We Like
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for pydantic_pkgr-0.1.2-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | dae5b26cd7d37e45fbd5c600c4e2ea874ad17a6ad0af8b7ee1daceb02fe67038 |
|
| MD5 | 015eae8270289f1b683daf70377b8071 |
|
| BLAKE2b-256 | 58aaa23d88183dce30a828e40ce2697247c3ccf48c9c9d30730274f01656744e |
