Skip to main content

abstract_pypit

One-command PyPI publisher + GitHub pusher, a server-wide staging release loop (init / stage / release / request / runner), a Postgres mirror of package trees (src) and fleet distribution to hosts (fleet).

Finds the next free version above whatever is on PyPI, bumps setup.py and pyproject.toml, builds sdist + wheel, uploads to PyPI via twine, commits and pushes to GitHub, then syncs the local install — all in one call.

Dependencies: requests, plus build and twine (so a fresh install can publish).

Install

pip install abstract_pypit

Usage

# from any package directory that has setup.py + pyproject.toml:
from pypit import runit
runit()
# or from the command line:
abstract-pypit

Staging release loop

The source of a package is a directory on the server (default /srv/pyit/dev/<project>, symlinks resolved). Nobody edits it directly; changes reach it only through a disposable staging dir and a host-side runner:

# once, as the user who owns the staging root (the "runner"):
abstract-pypit runner install --root /srv/vm_mgr/src --requesters vm_mgr,hugpy
abstract-pypit init abstract_utilities --root /srv/vm_mgr/src      # staging dir + push.sh, then stage

# any requester:
/srv/vm_mgr/src/abstract_utilities/push.sh stage     # source -> ./source -> ./live
#   ... edit ./live, test there ...
/srv/vm_mgr/src/abstract_utilities/push.sh release   # upload, then the source is swapped
/srv/vm_mgr/src/abstract_utilities/push.sh           # status

push.sh only drops <root>/_pypit/requests/<project>.<action>.<id>.req and follows the log (abstract-pypit request ...); the owner's systemd --user path unit runs abstract-pypit runner run, which executes requests one at a time. No ssh, sudo or credentials are needed to release.

release, per project:

  1. If the source moved on since staging (version, content or git HEAD), it PAUSES: writes merge-*.diff files, 3-way merges the source into live/ (base = source/), marks conflicts <<<<<<< live / ======= / >>>>>>> source (or a <file>.CONFLICT note; nothing wins), rebases source/ and stops. The merged live is what the next push releases; it refuses while markers or notes remain.
  2. Otherwise, by backend (auto-detected):
    • pypi: runPypit inside live/ (next free version, build, upload, commit+push without pulling, sync the local install); success = PyPI lists the version.
    • pkg_src: the source is <repo>/py/<group>/<pkg> of a pkg-src-watched tree (<repo>/py/tooling/pkg_src.py); the swap is the release; success = pkg-src-watch recorded exactly the swapped tree (read-only query), else the swap is reversed.
  3. live/ moves to prod/, prod/ is copied to a temp dir and content-verified, then source -> _old, temp -> source, _old removed; then it re-stages.

Copies are rsync -aAX --checksum, keep .git, and exclude build output. Commands:

command
abstract-pypit init <p> [--root R] [--source S] [--editors u1,u2] [--handoff R2] [--force] create/rebuild the staging dir, stage
abstract-pypit stage <p> [--force] / release <p> / status <p> the engine (run by the runner)
abstract-pypit request <stage|release|status> <p> what push.sh calls
abstract-pypit runner install [--root R ...] [--requesters u1,u2] / run --root R / status the runner
abstract-pypit src ... / abstract-pypit fleet ... the mirror and the fleet (below)

Env: PYPIT_ROOT (default root ~/staging), PYPIT_DEV_ROOT (default /srv/pyit/dev), PYPIT_PYPI_WAIT, PYPIT_PKG_SRC_WAIT, PYPIT_PKG_SRC_PYTHON. Tests: python -m unittest discover -s tests (nothing is uploaded).

Package mirror + fleet (src, fleet)

hugpy's pkg-src-watch system, housed here and generalized to any tree, DB and schema. pip install 'abstract-pypit[src]' adds psycopg (Postgres); the fleet hub and agent need only the base package.

src — mirror a tree of packages into Postgres. Every package (a directory with a pyproject.toml, found by globs; hugpy's */*/pyproject.toml layout is detected first) gets a table <schema>.pkg_<package>, one row per file, so a build recreates it from the table. Each distinct tree is an immutable version (contents deduped by sha256; identity = path + content + exec bit), labelled <release>.<micro>: a tag's tree is 0.2.1, later trees 0.2.1.1, 0.2.1.2 … resetting when the release advances. Configurations pin one version per package. good / config good refuse unless a verify job passed on exactly that version / member set, in a hashed test env that matched its fingerprint (optionally through a sandbox launcher). The schema is abstract-pypit's own (default pypit_src); a schema it did not create is refused.

abstract-pypit src init hugpy --tree /path/to/tree --dsn 'dbname=mydb' [--schema pypit_src]
abstract-pypit src sync && abstract-pypit src verify     # mirror; rebuild + compare byte-for-byte
abstract-pypit src install                               # the watch: systemd --user unit
abstract-pypit src status | versions | config save NAME | config diff A B
abstract-pypit src job verify PKG@REF [--now]  ->  abstract-pypit src good PKG REF
abstract-pypit src restore PKG REF [--apply]   |   config restore NAME [--apply]

The watch polls every interval s, syncs a package once it has been quiet for settle s, runs queued verify/restore jobs one at a time, optionally pins each settled edit as config dev-<utc> (--auto-config) and steps the fleets bound to the mirror.

fleet — ship configurations to hosts and keep them converged. A configuration becomes a release: one wheel per package built from the DB trees at one lockstep <base>.postN (or, --scheme label, each package at its own label), checked (versions, sibling pins), published immutably into an index dir. A pin (<index>/fleet.json + constraints.txt) points the fleet at a release; the hub serves the dir as a PEP 503 index and takes host reports; each host's agent pip-installs the pinned set under those constraints, runs its restart commands, checks health and reports. Each pin opens a rollout judged from those reports (healthy / unhealthy / unjudged; unhealthy releases are never auto-re-pushed). Rollback = pin an older release.

abstract-pypit fleet init prod --mirror hugpy --index-dir /srv/wheels [--gate known-good|none] [--auto]
abstract-pypit fleet publish CONFIG --pin       # or let the watch do it (--auto)
abstract-pypit fleet hub install                # serve the index + pin (default 127.0.0.1:9300)
abstract-pypit fleet status | releases | tick | pin RELEASE
# on every host (needs only `pip install abstract-pypit`):
abstract-pypit fleet agent init app --hub http://hub:9300 --python /srv/app/venv/bin/python \
    --track my-dist --restart 'systemctl --user restart app.service' --health-url http://127.0.0.1:8000/health
abstract-pypit fleet agent install app

Settings live in $PYPIT_CONFIG_DIR (default ~/.config/abstract-pypit): src/<mirror>.json, fleet/<fleet>.json, agent/<agent>.json. Tests: PYPIT_TEST_DSN='dbname=…' python -m unittest discover -s tests (DB tests use a throwaway schema and drop it; without the variable they are skipped).

Credentials

PyPI: twine reads ~/.pypirc or TWINE_USERNAME / TWINE_PASSWORD env vars.

GitHub: create pypit/src/envs/.env on the machine running pypit:

GITHUB_OWNER_1=your-username
GITPASS_1=<your-github-pat>
GITHUB_OWNER_2=your-org
GITPASS_2=<org-github-pat>

SSH key at ~/.ssh/github/githubssh_nopass must be registered with GitHub.

Per-package config

Add to the package's own pyproject.toml:

[tool.pypit]
github_owner = "your-org-or-username"   # which org/user owns the repo
github_push  = true                      # set false to skip GitHub entirely

License

MIT

Metadata

Release files for abstract-pypit 0.1.13

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for abstract-pypit 0.1.13
File Size Uploaded Provenance Actions
abstract_pypit-0.1.13.tar.gz 142.1 kB Without provenance Details for abstract_pypit-0.1.13.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for abstract-pypit 0.1.13
File Interpreter ABI Platform Actions
abstract_pypit-0.1.13-py3-none-any.whl Python 3 none any Details for abstract_pypit-0.1.13-py3-none-any.whl

Total release size: 283.9 kB

Release files / abstract_pypit-0.1.13.tar.gz

Download URL abstract_pypit-0.1.13.tar.gz
Size 142.1 kB
Tags Source
SHA-256 checksum
How to use checksums
ec40b8f637beedf0fee3a41d2da2dbf7f87429cfe9eff14333cf5897d6a3dbcd
BLAKE2b-256 checksum
How to use checksums
a00b6efe5ac7e0c9e9fc94759d6dc8489f81c69d965f5614e0844b7c651118d2
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Release files / abstract_pypit-0.1.13-py3-none-any.whl

Download URL abstract_pypit-0.1.13-py3-none-any.whl
Size 141.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
98a34e9a8692f81d7baab97dac5c9a53a0f785ca2cc7fabd94addd7dd5a4b7a2
BLAKE2b-256 checksum
How to use checksums
5d52c9096ca08ac563c0316f0471ceb9003e50ee4cb48334f84887e798698fc6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.12.3

Security for abstract-pypit 0.1.13

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page