dkdc

don't know, don't care - A modular data lake management system with encryption, virtual filesystem, and development tools.

Features

🔐 Encrypted Storage - All data encrypted at rest using DuckLake
📁 Virtual Filesystem - Store and organize files in a virtual filesystem
🔑 Secrets Management - Secure storage for API keys and passwords
📦 Directory Archiving - Archive directories with gitignore support
🚀 Development REPL - Interactive Python/SQL environment for data exploration
🦀 Written in Rust - Fast, safe, and reliable

Installation

Prerequisites

uv - For Python environment management
Rust - For building from source (optional)

Quick Install

# Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install dkdc as a tool (from PyPI)
uv tool install dkdc

# Or install from git
uv tool install git+https://github.com/lostmygithubaccount/dkdc

Development Install

# Clone the repository
git clone https://github.com/lostmygithubaccount/dkdc.git
cd dkdc

# Create virtual environment
uv venv

# Build the Rust extensions and install in development mode
uv run maturin develop

# Now you can use dkdc
uv run dkdc --help

[!NOTE] After making changes to Rust code, run uv run maturin develop to rebuild.

Usage

Development REPL

Enter an interactive Python environment with DuckDB and ibis pre-configured:

# Python REPL (default)
dkdc dev

# SQL REPL
dkdc dev --sql

Files Management

Work with files in the virtual filesystem:

# Add a file
dkdc files add README.md

# List files
dkdc files list

# Open a file
dkdc files open README.md

# Dump all files to local directory
dkdc files dump ./output

# Restore files from directory
dkdc files restore ./backup

Secrets Management

Securely store and retrieve secrets:

# Set a secret (interactive)
dkdc secrets set API_KEY

# Set a secret directly
dkdc secrets set API_KEY --value "your-secret-value"

# Get a secret
dkdc secrets get API_KEY

# List all secrets
dkdc secrets list

# Export secrets
dkdc secrets export              # Shell format to stdout
dkdc secrets export -f json       # JSON format
dkdc secrets export -f dotenv     # .env format
dkdc secrets export .env          # Write to file

Archive Directories

Archive directories to the data lake:

# Archive current directory
dkdc archive

# Archive specific directory
dkdc archive /path/to/project

# Archive with custom name
dkdc archive /path/to/project --name backup-2024.zip

Configuration

Edit the configuration file:

dkdc --config

Configuration is stored at ~/.config/dkdc/config.toml.

Architecture

dkdc uses a modular architecture where each major feature is implemented as a separate Rust crate:

dkdc-config - Configuration management
dkdc-lake - Core DuckDB/DuckLake functionality
dkdc-dev - Development REPL
dkdc-files - Virtual filesystem
dkdc-secrets - Secrets management
dkdc-archive - Directory archiving
dkdc-cli - Main CLI application

See ARCHITECTURE.md for detailed technical documentation.

Data Storage

All data is stored encrypted using DuckDB with the DuckLake extension:

Location: ~/.dkdc/dkdclake/
Metadata: SQLite database at ~/.dkdc/dkdclake/metadata.db
Data: Encrypted files in ~/.dkdc/dkdclake/data/

Development

Setup Development Environment

# Clone the repository
gh repo clone lostmygithubaccount/dkdc
cd dkdc

# Set up the development environment
./bin/setup.sh

# Build the Rust extensions (required for first use)
uv run maturin develop

# Run code checks (lint & format)
./bin/check.sh

# Run the development REPL
uv run dkdc dev

Organization

py/: Python source code (thin wrapper)
rs/: Rust source code (main implementation)
bin/: Repository utilities
tasks/: Development tasks
dotfiles/: Dotfiles
websites/: Static websites

Development Workflow

Make changes to Rust code
Run uv run maturin develop to rebuild
Test with uv run dkdc

Building for Release

# Build wheel for distribution
uv run maturin build --release

# The wheel will be in target/wheels/
ls target/wheels/

Security

All data is encrypted at rest using DuckLake
Secrets are stored as encrypted BLOBs
No secrets are logged or displayed unless explicitly requested
Virtual environment is isolated in ~/.dkdc/venv/

Roadmap

Planned Features

Backup/Restore - Cloud backup with triple-layer encryption
Data Versioning - Track changes to files over time
Access Control - User permissions and sharing
Cloud Storage - S3/GCS/Azure backend support
Data Lineage - Track data transformations

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

MIT License - see LICENSE for details.

Acknowledgments

Built with:

DuckDB - In-process analytical database
DuckLake - Encryption extension for DuckDB
Rust - Systems programming language
uv - Fast Python package manager

Development Status
- 3 - Alpha
License
- OSI Approved :: MIT License
Programming Language
- Python :: 3.13
- Rust

Metadata

Development Status
- 3 - Alpha
License
- OSI Approved :: MIT License
Programming Language
- Python :: 3.13
- Rust

Release files for dkdc 0.50.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for dkdc 0.50.0
File	Interpreter	ABI	Platform	Reset
dkdc-0.50.0-cp313-cp313-macosx_11_0_arm64.whl 10.3 MB Jul 2, 2025	CPython 3.13	CPython 3.13	macOS 11.0+ ARM64	Details

Release files / dkdc-0.50.0-cp313-cp313-macosx_11_0_arm64.whl

Download URL	dkdc-0.50.0-cp313-cp313-macosx_11_0_arm64.whl
Size	10.3 MB
Tags	CPython 3.13 macOS 11.0+ ARM64
SHA-256 checksum How to use checksums	`8b6c67523a2bf140b854512e63009b3968b1ac792ee669a43797e23d25582012`
BLAKE2b-256 checksum How to use checksums	`53dd28ab5ec2b78f6f1bce6897939fcb012cf7f28c0a4672601f12ecd6924b66`
Upload date	Jul 2, 2025
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`uv/0.7.18`