Big file collection manager
Project description
pyfiles
A Big file collection manager.
Install
In a virtual env:
python3 -m venv .venv
source .venv/bin/activate
pip install -U pip wheel
pip install git+git://github.com/jrmi/pyfiles@master # Or any last commit
# Then create your setup.py file before using CLI
CLI
Create a settings.py files where you want to execute the cli with this configuration for file storage:
BACKEND = "pyfiles.storages.diskstorage.DiskStorage"
BACKEND_OPTIONS = {
"basepath": "/tmp/tmpdir",
"base_url": "http://localhost:8000"
}
And for S3:
BACKEND = "pyfiles.storages.s3storage.S3Storage"
BACKEND_OPTIONS = {
"access_key":"<you-S3-access-key>",
"secret_key":"<you-S3-secret-key>",
"endpoint_url":"<S3-api-endpoint>",
"region_name":"<region>",
"bucket_name":"<bucket name>",
}
Then to store a file:
$ pyfiles store <file path> <file.namespace> <file.name> <version>
version should respect the format: YYYY.MM.DD-Rev or any semver like X.Y.Z
To list all version of a file:
$ pyfiles versions <file.namespace> <file.name>
To search for a file:
$ pyfiles search <file.namespace> <file.name> [<version-prefix>]
version-prefix can be YYYY or X or YYYY.MM or X.Y or YYYY.MM.DD or X.Y.Z or Latest. Latest by default if missing.
Finnaly to delete a file:
$ pyfiles delete <file.namespace> <file.name> <version>
To start the web api server:
$ pyfiles serve
Web API
GET on /search/<namespace>/<filename>[?version=<version>]
To get file version download link. Namespace is a namespace to organise data and filename is the file name. You can optionnaly add a version like latest or <year> or <year.month> or <major> or <major>.<minor>, … You get the latest for the specified version.
GET on /versions/<namespace>/<filename>
To show all avaible file versions.
Python API
See pyfiles.storage classes for more informations.
You can use pyfiles.storage.get_storage(<backend path>, <options>) to initialize your storage.
Features
An API to download files with rich version selection
List all version of a file
Can be used for CSV or Geojson files
File can have version like 2018.01.10-01
Find file by a part of the version. 2018 or 2018.01
Roadmap
Allow authentification with private data
Handle file diff between versions
Get the update date of a file to ease caching
Add a client library and CLI
License
Free software: MIT license
Documentation: https://pyfiles.readthedocs.io.
Credits
This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.
History
0.1.1 (2020-12-17)
First release on PyPI.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.