Datamate
Datamate is a data and configuration management framework in Python for machine-learning research. It uses the filesystem as memory through Directory objects, providing a programming interface to store and retrieve files in hierarchical structures using HDF5.
Main Features
- Filesystem as memory through Directory objects
- Hierarchical data organization
- Automatic path handling and resolution with pathlib
- Array storage in HDF5 format
- Parallel read/write operations
- Configuration-based compilation and access of data
- Configuration management in YAML files
- Configuration comparison and diffing
- Pandas DataFrame integration
- Directory structure visualization (tree view)
- Experiment status tracking
Example
import datamate
import numpy as np
# Set up experiment directory
datamate.set_root_dir("./experiments")
# Set up experiment configuration
config = {
"model": "01",
"optimizer": "Adam",
"learning_rate": 0.001,
"n_epochs": 100
}
# Set up experiment directory and store configuration
exp = datamate.Directory("vision_study/model_01", config)
# Store arrays as HDF5 files
exp.images = np.random.rand(100, 64, 64) # stored as images.h5
exp.responses = np.zeros((100, 1000)) # stored as responses.h5
# Access data
mean_response = exp.responses[:].mean()
More detailed examples in the documentation.
Installation
Using pip:
pip install datamate
Documentation
Full documentation is available at flyvis.github.io/datamate.
Related Projects
Contributing
Contributions welcome! Please check our Contributing Guide for guidelines.
1 maintainer
Metadata
1 maintainer
Release files for datamate 1.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| datamate-1.0.0.tar.gz | 463.3 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| datamate-1.0.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 496.9 kB
Release files / datamate-1.0.0.tar.gz
| Download URL | datamate-1.0.0.tar.gz |
|---|---|
| Size | 463.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
87a978514b1dadfbd88284733f5f6d02578ade2c29b92cfa93f46e93a2f344c7
|
|
BLAKE2b-256 checksum How to use checksums |
1bc1a9fa4cd2c560cbd3a7c7250f7841055fd88c4e1e35fdec1357662336d624
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.12.2
|
Release files / datamate-1.0.0-py3-none-any.whl
| Download URL | datamate-1.0.0-py3-none-any.whl |
|---|---|
| Size | 33.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
0eb22107785354b81563915cfd3c946ee933c2a7bc8fc9af935cc2fe5494a5ab
|
|
BLAKE2b-256 checksum How to use checksums |
89aa8a7bf0eea47a45edee6a3cbdfd95c5b4ad3a699003b029c6f63efc1bc1ab
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.12.2
|
