abstract_pypit

One-command PyPI publisher + GitHub pusher, a server-wide staging release loop (init / stage / release / request / runner), a Postgres mirror of package trees (src) and fleet distribution to hosts (fleet).

Finds the next free version above whatever is on PyPI, bumps setup.py and pyproject.toml, builds sdist + wheel, uploads to PyPI via twine, commits and pushes to GitHub, then syncs the local install — all in one call.

Dependencies: requests, plus build and twine (so a fresh install can publish).

Install

pip install abstract_pypit

Usage

# from any package directory that has setup.py + pyproject.toml:
from pypit import runit
runit()

# or from the command line:
abstract-pypit

Staging release loop

The source of a package is a directory on the server (default /srv/pyit/dev/<project>, symlinks resolved). Nobody edits it directly; changes reach it only through a disposable staging dir and a host-side runner:

# once, as the user who owns the staging root (the "runner"):
abstract-pypit runner install --root /srv/vm_mgr/src --requesters vm_mgr,hugpy
abstract-pypit init abstract_utilities --root /srv/vm_mgr/src      # staging dir + push.sh, then stage

# any requester:
/srv/vm_mgr/src/abstract_utilities/push.sh stage     # source -> ./source -> ./live
#   ... edit ./live, test there ...
/srv/vm_mgr/src/abstract_utilities/push.sh release   # upload, then the source is swapped
/srv/vm_mgr/src/abstract_utilities/push.sh           # status

push.sh only drops <root>/_pypit/requests/<project>.<action>.<id>.req and follows the log (abstract-pypit request ...); the owner's systemd --user path unit runs abstract-pypit runner run, which executes requests one at a time. No ssh, sudo or credentials are needed to release.

release, per project:

1. If the source moved on since staging (version, content or git HEAD), it PAUSES: writes merge-*.diff files, 3-way merges the source into live/ (base = source/), marks conflicts <<<<<<< live / ======= / >>>>>>> source (or a <file>.CONFLICT note; nothing wins), rebases source/ and stops. The merged live is what the next push releases; it refuses while markers or notes remain.
2. Otherwise, by backend (auto-detected):
   • pypi: runPypit inside live/ (next free version, build, upload, commit+push without pulling, sync the local install); success = PyPI lists the version.
   • pkg_src: the source is <repo>/py/<group>/<pkg> of a pkg-src-watched tree (<repo>/py/tooling/pkg_src.py); the swap is the release; success = pkg-src-watch recorded exactly the swapped tree (read-only query), else the swap is reversed.
3. live/ moves to prod/, prod/ is copied to a temp dir and content-verified, then source -> _old, temp -> source, _old removed; then it re-stages.

Copies are rsync -aAX --checksum, keep .git, and exclude build output. Commands:

Env: PYPIT_ROOT (default root ~/staging), PYPIT_DEV_ROOT (default /srv/pyit/dev), PYPIT_PYPI_WAIT, PYPIT_PKG_SRC_WAIT, PYPIT_PKG_SRC_PYTHON. Tests: python -m unittest discover -s tests (nothing is uploaded).

Package mirror + fleet (src, fleet)

hugpy's pkg-src-watch system, housed here and generalized to any tree, DB and schema. pip install 'abstract-pypit[src]' adds psycopg (Postgres); the fleet hub and agent need only the base package.

src — mirror a tree of packages into Postgres. Every package (a directory with a pyproject.toml, found by globs; hugpy's */*/pyproject.toml layout is detected first) gets a table <schema>.pkg_<package>, one row per file, so a build recreates it from the table. Each distinct tree is an immutable version (contents deduped by sha256; identity = path + content + exec bit), labelled <release>.<micro>: a tag's tree is 0.2.1, later trees 0.2.1.1, 0.2.1.2 … resetting when the release advances. Configurations pin one version per package. good / config good refuse unless a verify job passed on exactly that version / member set, in a hashed test env that matched its fingerprint (optionally through a sandbox launcher). The schema is abstract-pypit's own (default pypit_src); a schema it did not create is refused.

abstract-pypit src init hugpy --tree /path/to/tree --dsn 'dbname=mydb' [--schema pypit_src]
abstract-pypit src sync && abstract-pypit src verify     # mirror; rebuild + compare byte-for-byte
abstract-pypit src install                               # the watch: systemd --user unit
abstract-pypit src status | versions | config save NAME | config diff A B
abstract-pypit src job verify PKG@REF [--now]  ->  abstract-pypit src good PKG REF
abstract-pypit src restore PKG REF [--apply]   |   config restore NAME [--apply]

The watch polls every interval s, syncs a package once it has been quiet for settle s, runs queued verify/restore jobs one at a time, optionally pins each settled edit as config dev-<utc> (--auto-config) and steps the fleets bound to the mirror.

fleet — ship configurations to hosts and keep them converged. A configuration becomes a release: one wheel per package built from the DB trees at one lockstep <base>.postN (or, --scheme label, each package at its own label), checked (versions, sibling pins), published immutably into an index dir. A pin (<index>/fleet.json + constraints.txt) points the fleet at a release; the hub serves the dir as a PEP 503 index and takes host reports; each host's agent pip-installs the pinned set under those constraints, runs its restart commands, checks health and reports. Each pin opens a rollout judged from those reports (healthy / unhealthy / unjudged; unhealthy releases are never auto-re-pushed). Rollback = pin an older release.

abstract-pypit fleet init prod --mirror hugpy --index-dir /srv/wheels [--gate known-good|none] [--auto]
abstract-pypit fleet publish CONFIG --pin       # or let the watch do it (--auto)
abstract-pypit fleet hub install                # serve the index + pin (default 127.0.0.1:9300)
abstract-pypit fleet status | releases | tick | pin RELEASE
# on every host (needs only `pip install abstract-pypit`):
abstract-pypit fleet agent init app --hub http://hub:9300 --python /srv/app/venv/bin/python \
    --track my-dist --restart 'systemctl --user restart app.service' --health-url http://127.0.0.1:8000/health
abstract-pypit fleet agent install app

Settings live in $PYPIT_CONFIG_DIR (default ~/.config/abstract-pypit): src/<mirror>.json, fleet/<fleet>.json, agent/<agent>.json. Tests: PYPIT_TEST_DSN='dbname=…' python -m unittest discover -s tests (DB tests use a throwaway schema and drop it; without the variable they are skipped).

Credentials

PyPI: twine reads ~/.pypirc or TWINE_USERNAME / TWINE_PASSWORD env vars.

GitHub: create pypit/src/envs/.env on the machine running pypit:

GITHUB_OWNER_1=your-username
GITPASS_1=<your-github-pat>
GITHUB_OWNER_2=your-org
GITPASS_2=<org-github-pat>

SSH key at ~/.ssh/github/githubssh_nopass must be registered with GitHub.

Per-package config

Add to the package's own pyproject.toml:

[tool.pypit]
github_owner = "your-org-or-username"   # which org/user owns the repo
github_push  = true                      # set false to skip GitHub entirely

License

MIT