Simple, file-based data version control
Project description
danvers: simple file-based data version control
Danvers is a Python data version management tool, which helps you to maintain and reference current and previous versions of data files.
This means you can ensure code and models run consistently when they reference datasets that may change and that you can ensure you keep older copies of data files used as part of decision making, so logic and code can be rerun.
New versions are only created if the new file is different to previous versions. This allows infrequently updated data sets to be easily maintained and versioned - if it's new, Danvers will keep a copy, if it hasn't changed, it will ignore and keep its current version.
Features
Danvers is simple, rather than feature-rich, but here are some of the things it can do:
- Access previous versions of data files
- Maintain a fixed number of verions (or all versions)
- Different trimming strategies are available (first-in-first-out, last-used-first-out)
- Automatic duplicate check for data files against all known versions
Dependencies
Danvers has no dependencies, just what comes with Python.
Example Usage
from danvers import Danvers
# instantiate with the location the data is stored
vers = Danvers(r'data')
# create the dataset if it doesn't exist already
if not 'marvel_movies' in vers.read_datasets():
vers.create_dataset('marvel_movies')
# add the first data file, should return verion 1
version = vers.create_data_file('marvel_movies', r'test_data\movies_phase_1.csv')
print(version)
# adding a new data file should return version 2
version = vers.create_data_file('marvel_movies', r'test_data\movies_phase_1+2.csv')
print(version)
# get the filename for the latest version of the data
filename = vers.get_data_file('marvel_movies')
print(filename)
# get the filename for version 1 of the data
filename = vers.get_data_file('marvel_movies', 1)
print(filename)
License
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.