dkdc
don't know, don't care - A modular data lake management system with encryption, virtual filesystem, and development tools.
Features
- 🔐 Encrypted Storage - All data encrypted at rest using DuckLake
- 📁 Virtual Filesystem - Store and organize files in a virtual filesystem
- 🔑 Secrets Management - Secure storage for API keys and passwords
- 📦 Directory Archiving - Archive directories with gitignore support
- 🚀 Development REPL - Interactive Python/SQL environment for data exploration
- 🦀 Written in Rust - Fast, safe, and reliable
Installation
Prerequisites
Quick Install
# Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh
# Install dkdc as a tool (from PyPI)
uv tool install dkdc
# Or install from git
uv tool install git+https://github.com/lostmygithubaccount/dkdc
Development Install
# Clone the repository
git clone https://github.com/lostmygithubaccount/dkdc.git
cd dkdc
# Create virtual environment
uv venv
# Build the Rust extensions and install in development mode
uv run maturin develop
# Now you can use dkdc
uv run dkdc --help
Usage
Development REPL
Enter an interactive Python environment with DuckDB and ibis pre-configured:
# Python REPL (default)
dkdc dev
# SQL REPL
dkdc dev --sql
Files Management
Work with files in the virtual filesystem:
# Add a file
dkdc files add README.md
# List files
dkdc files list
# Open a file
dkdc files open README.md
# Dump all files to local directory
dkdc files dump ./output
# Restore files from directory
dkdc files restore ./backup
Secrets Management
Securely store and retrieve secrets:
# Set a secret (interactive)
dkdc secrets set API_KEY
# Set a secret directly
dkdc secrets set API_KEY --value "your-secret-value"
# Get a secret
dkdc secrets get API_KEY
# List all secrets
dkdc secrets list
# Export secrets
dkdc secrets export # Shell format to stdout
dkdc secrets export -f json # JSON format
dkdc secrets export -f dotenv # .env format
dkdc secrets export .env # Write to file
Archive Directories
Archive directories to the data lake:
# Archive current directory
dkdc archive
# Archive specific directory
dkdc archive /path/to/project
# Archive with custom name
dkdc archive /path/to/project --name backup-2024.zip
Configuration
Edit the configuration file:
dkdc --config
Configuration is stored at ~/.config/dkdc/config.toml.
Architecture
dkdc uses a modular architecture where each major feature is implemented as a separate Rust crate:
- dkdc-config - Configuration management
- dkdc-lake - Core DuckDB/DuckLake functionality
- dkdc-dev - Development REPL
- dkdc-files - Virtual filesystem
- dkdc-secrets - Secrets management
- dkdc-archive - Directory archiving
- dkdc-cli - Main CLI application
See ARCHITECTURE.md for detailed technical documentation.
Data Storage
All data is stored encrypted using DuckDB with the DuckLake extension:
- Location:
~/.dkdc/dkdclake/ - Metadata: SQLite database at
~/.dkdc/dkdclake/metadata.db - Data: Encrypted files in
~/.dkdc/dkdclake/data/
Development
Setup Development Environment
# Clone the repository
gh repo clone lostmygithubaccount/dkdc
cd dkdc
# Set up the development environment
./bin/setup.sh
# Build the Rust extensions (required for first use)
uv run maturin develop
# Run code checks (lint & format)
./bin/check.sh
# Run the development REPL
uv run dkdc dev
Organization
py/: Python source code (thin wrapper)rs/: Rust source code (main implementation)bin/: Repository utilitiestasks/: Development tasksdotfiles/: Dotfileswebsites/: Static websites
Development Workflow
- Make changes to Rust code
- Run
uv run maturin developto rebuild - Test with
uv run dkdc
Building for Release
# Build wheel for distribution
uv run maturin build --release
# The wheel will be in target/wheels/
ls target/wheels/
Security
- All data is encrypted at rest using DuckLake
- Secrets are stored as encrypted BLOBs
- No secrets are logged or displayed unless explicitly requested
- Virtual environment is isolated in
~/.dkdc/venv/
Roadmap
Planned Features
- Backup/Restore - Cloud backup with triple-layer encryption
- Data Versioning - Track changes to files over time
- Access Control - User permissions and sharing
- Cloud Storage - S3/GCS/Azure backend support
- Data Lineage - Track data transformations
Contributing
Contributions are welcome! Please feel free to submit a Pull Request.
License
MIT License - see LICENSE for details.
Acknowledgments
Built with:
Metadata
Release files for dkdc 0.50.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| dkdc-0.50.0-cp313-cp313-macosx_11_0_arm64.whl | CPython 3.13 | CPython 3.13 | macOS 11.0+ ARM64 | Details for dkdc-0.50.0-cp313-cp313-macosx_11_0_arm64.whl |
Release files / dkdc-0.50.0-cp313-cp313-macosx_11_0_arm64.whl
| Download URL | dkdc-0.50.0-cp313-cp313-macosx_11_0_arm64.whl |
|---|---|
| Size | 10.3 MB |
| Tags | CPython 3.13 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
8b6c67523a2bf140b854512e63009b3968b1ac792ee669a43797e23d25582012
|
|
BLAKE2b-256 checksum How to use checksums |
53dd28ab5ec2b78f6f1bce6897939fcb012cf7f28c0a4672601f12ecd6924b66
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.7.18
|
Security for dkdc 0.50.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.