etcetera
Dataset sharing via cloud storage (S3, Google Storage)
Mental model
A dataset is an immutable collection of files organized in directories (e.g train/, val/).
A dataset can have a meta.json file, which is a collection of arbitraty key/value pairs.
Dataset can be local or remote. Local datasets are kept in ~/.etc/. Remote datasets are tgz files stored in cloud storage.
PyPI package etcetera provides:
- a command-line utility
etc - Python package
etcetera
Using Command Line
etc -h
usage: etc [-h] {ls,register,pull,push,purge} ...
etcetera: managing cloud-hosted datasets
positional arguments:
{ls,register,pull,push,purge}
command
ls List datasets
register Register directory as a dataset
pull Pull dataset from repository
push Push dataset to the repository
purge Purge local dataset
optional arguments:
-h, --help show this help message and exit
Using Python
import etcetera as etc
dataset = etc.dataset('flower', auto_install=True)
dataset.keys()
>> { 'test', 'train' }
for filename in dataset['train'].iterdir():
print(filename)
>> "~/.etc/flower/train/data00001.txt"
>> "~/.etc/flower/train/data00002.txt"
dataset.meta
>> {}
dataset.root
>> "~/.etc/flower"
Installing
pip install 'etcetera[s3]'
Installs etceters with the support for S3 cloud.
Configuration
~/.etc.toml contains configuration for the service in TOML format. Example:
url = "s3://my-bucket"
Another example:
url = "s3://my-bucket"
public = false
aws_access_key_id = "Axxxx"
aws_secret_access_key = "Kxxx"
endpoint_url = "https://s3.amazonaws.com"
A configuration file is required for remote operations (pull, push, ls -r). It is not required for local operations (ls, register).
In configuration file url value is required. The rest is optional.
url: URL of the remote repository. For example,s3://my-bucket.public: set totrueif you wantpushto create publicly-readable cloud files. Default isfalse.aws_access_key_id,aws_secret_access_key,endpoint_url: configuration files to access AWS api. If not set, the defaults from global AWS config will be used.
Command-line example
etc ls
etc ls -r
etc pull MNIST
etc register <directory> as SuperMNIST
Creating a dataset
A dataset must have:
datadirectory (non-empty)datadirectory must not have any files, only sub-directories (we call them "partitions")
Optional:
meta.jsonREADME.md- other sub-directories, for example
assets/
A minimal dataset example
sample/
data/
train/
data00001.json
data00002.json
data00003.json
A general dataset example
sample/
README.md
meta.json
assets/
Analysis.ipynb
DataCleanup.ipynb
data/
train/
data00001.json
...
test/
test00001.json
...
val/
val00001.json
...
Metadata
Release files for etcetera 0.0.12
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| etcetera-0.0.12-py3-none-any.whl | Python 3 | none | any | Details for etcetera-0.0.12-py3-none-any.whl |
Release files / etcetera-0.0.12-py3-none-any.whl
| Download URL | etcetera-0.0.12-py3-none-any.whl |
|---|---|
| Size | 9.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b19fafbe904d5f131e60fa68005b935050a71505622dcae062a6f331ed3d487a
|
|
BLAKE2b-256 checksum How to use checksums |
acd6a3f1c4e0555f9d02b6cc390ec99d67e33d4fdec2961a59ef42ffb9647729
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.1.1 pkginfo/1.5.0.1 requests/2.24.0 setuptools/41.4.0 requests-toolbelt/0.9.1 tqdm/4.46.1 CPython/3.8.0
|
Security for etcetera 0.0.12
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.