Core API for managing (scientific) data
Project description
Documentation: http://dtoolcore.readthedocs.io
Free software: MIT License
Features
Core API for adding different types of metadata to files on disk
Automatic generation of structural metadata
Programmatic discovery and access of items in a dataset
Structural metadata includes hash, size and modification time for subsequent integrity checks
Ability to annotate individual files with arbitrary metadata
Metadata stored on disk as plain text files, i.e. disk datasets generated using this API can be accessed without special tools
Ability to create plugins for custom storage solutions
Plugins for iRODS and Microsoft Azure storage backends available
Cross-platform: Linux, Mac and Windows are all supported
Works with Python 2.7, 3.5 and 3.6
No external dependencies
Overview
The dtoolcore project provides a Python API for managing (scientific) data. It aims to help in three areas:
Adding structure and meta data to your project and files
Providing programmatic discovery of your data
Verifying the integrity of your data
Creating a custom storage plugin
Examine the code in dtoolcore.storagebroker.DiskStorageBroker.
Create a Python class for your storage, e.g. MyStorageBroker
Add a MyStorageBroker.key` attribute to the class, this key is used to lookup an appropriate storage broker when interacting with a dataset
Add a dtoolcore.FileHasher instance that matches the hashing algorithm used by your storage to your MyStorageBroker.hasher attribute
Add implementations for all the public functions in dtoolcore.storagebroker.DiskStorageBroker class to MyStorageBroker
Expose the MyStorageBroker class as a dtool.storage_broker entrypoint, e.g. add a section along the lines of the below to the setup.py file:
entry_points={ "dtool.storage_brokers": [ "MyStorageBroker=my_dtool_storage_plugin:MyStorageBroker", ], },
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.