dvc-pandas
Loading from persisted source manifests
For pinned sources, resolve metadata once with Git available and persist the Pydantic model as JSON:
from dvc_pandas import Repository
repo = Repository(repo_url, dvc_remote="storage")
repo.set_target_commit(commit_sha)
manifest = repo.get_manifest(["transport/activity"])
stored_json = manifest.model_dump_json()
A later process can download and load the same datasets without constructing a Git or DVC repository:
from dvc_pandas import DatasetLoader, RepositoryManifest
manifest = RepositoryManifest.model_validate_json(stored_json)
loader = DatasetLoader(cache_root="/cache/dataset-objects")
loader.prefetch(manifest.datasets.values()) # Download only; no Parquet decoding.
dataset = loader.load(manifest.datasets["transport/activity"])
assert dataset.manifest == manifest.datasets["transport/activity"]
Manifest transport currently supports local filesystem and S3 DVC remotes,
including legacy and current DVC object layouts. Other remotes continue through
the existing Git-backed loader, without manifest provenance. Object URLs are
resolved by DVC when exporting a manifest; the downloader does not guess the
remote layout. Manifests contain full commit IDs, file hashes, metadata and
non-secret endpoint/region settings. Credentials are supplied through the normal
S3 environment/provider chain or DatasetLoader(storage_options=...) using
fsspec options; they are not serialized. Local remote paths must remain accessible
on the machine loading the manifest.
Downloads are checked against the manifest MD5, locked per content hash, and
published atomically. Existing local content-addressed files are trusted.
cache_root is the directory containing the two-character hash subdirectories.
Repository-backed loads use DVC's current local object directory; standalone
loaders default to dataset-objects under the dvc-pandas user cache. Pass the same
cache root to share files between loaders.
Dataset.manifest records source provenance and is also preserved by copy();
it does not identify subsequent in-memory edits. Datasets constructed from a
DataFrame have no manifest unless one is explicitly supplied through
DatasetMeta. Effective loaded units and index columns retain the existing
fallback to metadata embedded in Parquet.
The manifest format is versioned. Repository manifests describe the requested subset of datasets, not necessarily the entire repository. Callers own persistence, refresh policy for moving references, and selection of runtime credentials.
Run the full test suite:
uv run pytest
Release files for dvc-pandas 0.4.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| dvc_pandas-0.4.0.tar.gz | 191.6 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| dvc_pandas-0.4.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 207.3 kB
Release files / dvc_pandas-0.4.0.tar.gz
| Download URL | dvc_pandas-0.4.0.tar.gz |
|---|---|
| Size | 191.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1929cd61e9130bc59d2654f026895c9efcd2684df6d36bc2fcd3ae5badc6681f
|
|
BLAKE2b-256 checksum How to use checksums |
243cfef668df7e6c3a9b8264181f314991e572e8bd0eaf22b390ad2ea53b16b5
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.5
|
Release files / dvc_pandas-0.4.0-py3-none-any.whl
| Download URL | dvc_pandas-0.4.0-py3-none-any.whl |
|---|---|
| Size | 15.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2044800fafca39dbcdb3a1ac3c97b121d2f0f78e4a36d3784ea606423e6f6c41
|
|
BLAKE2b-256 checksum How to use checksums |
f35163d5ce84316e2d0bd23aa7f5163ac9b249dc509b8e84e9bac6a1981051f9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.12.5
|
