Skip to main content

Big file collection manager

Project description

pyfiles

https://img.shields.io/pypi/v/pyfiles.svg https://img.shields.io/travis/jrmi/pyfiles.svg Documentation Status Updates

A Big file collection manager.

Install

In a virtual env:

python3 -m venv .venv
source .venv/bin/activate
pip install -U pip wheel
pip install git+git://github.com/jrmi/pyfiles@master # Or any last commit
# Then create your setup.py file before using CLI

CLI

Create a settings.py files where you want to execute the cli with this configuration for file storage:

BACKEND = "pyfiles.storages.diskstorage.DiskStorage"

BACKEND_OPTIONS = {
    "basepath": "/tmp/tmpdir",
    "base_url": "http://localhost:8000"
}

And for S3:

BACKEND = "pyfiles.storages.s3storage.S3Storage"

BACKEND_OPTIONS = {
    "access_key":"<you-S3-access-key>",
    "secret_key":"<you-S3-secret-key>",
    "endpoint_url":"<S3-api-endpoint>",
    "region_name":"<region>",
    "bucket_name":"<bucket name>",

}

Then to store a file:

$ pyfiles store <file path> <file.namespace> <file.name> <version>

version should respect the format: YYYY.MM.DD-Rev or any semver like X.Y.Z

To list all version of a file:

$ pyfiles versions <file.namespace> <file.name>

To search for a file:

$ pyfiles search <file.namespace> <file.name> [<version-prefix>]

version-prefix can be YYYY or X or YYYY.MM or X.Y or YYYY.MM.DD or X.Y.Z or Latest. Latest by default if missing.

Finnaly to delete a file:

$ pyfiles delete <file.namespace> <file.name> <version>

To start the web api server:

$ pyfiles serve

Web API

GET on /search/<namespace>/<filename>[?version=<version>]

To get file version download link. Namespace is a namespace to organise data and filename is the file name. You can optionnaly add a version like latest or <year> or <year.month> or <major> or <major>.<minor>, … You get the latest for the specified version.

GET on /versions/<namespace>/<filename>

To show all avaible file versions.

Python API

See pyfiles.storage classes for more informations.

You can use pyfiles.storage.get_storage(<backend path>, <options>) to initialize your storage.

Features

  • An API to download files with rich version selection

  • List all version of a file

  • Can be used for CSV or Geojson files

  • File can have version like 2018.01.10-01

  • Find file by a part of the version. 2018 or 2018.01

Roadmap

  • Allow authentification with private data

  • Handle file diff between versions

  • Get the update date of a file to ease caching

  • Add a client library and CLI

License

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

History

0.1.1 (2020-12-17)

  • First release on PyPI.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

bygfiles-0.1.1.tar.gz (20.8 kB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page