dkdc 0.50.0

don't know, don't care

dkdc

don't know, don't care - A modular data lake management system with encryption, virtual filesystem, and development tools.

Features

• 🔐 Encrypted Storage - All data encrypted at rest using DuckLake
• 📁 Virtual Filesystem - Store and organize files in a virtual filesystem
• 🔑 Secrets Management - Secure storage for API keys and passwords
• 📦 Directory Archiving - Archive directories with gitignore support
• 🚀 Development REPL - Interactive Python/SQL environment for data exploration
• 🦀 Written in Rust - Fast, safe, and reliable

Installation

Prerequisites

• uv - For Python environment management
• Rust - For building from source (optional)

Quick Install

# Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh

# Install dkdc as a tool (from PyPI)
uv tool install dkdc

# Or install from git
uv tool install git+https://github.com/lostmygithubaccount/dkdc

Development Install

# Clone the repository
git clone https://github.com/lostmygithubaccount/dkdc.git
cd dkdc

# Create virtual environment
uv venv

# Build the Rust extensions and install in development mode
uv run maturin develop

# Now you can use dkdc
uv run dkdc --help

[!NOTE] After making changes to Rust code, run uv run maturin develop to rebuild.

Usage

Development REPL

Enter an interactive Python environment with DuckDB and ibis pre-configured:

# Python REPL (default)
dkdc dev

# SQL REPL
dkdc dev --sql

Files Management

Work with files in the virtual filesystem:

# Add a file
dkdc files add README.md

# List files
dkdc files list

# Open a file
dkdc files open README.md

# Dump all files to local directory
dkdc files dump ./output

# Restore files from directory
dkdc files restore ./backup

Secrets Management

Securely store and retrieve secrets:

# Set a secret (interactive)
dkdc secrets set API_KEY

# Set a secret directly
dkdc secrets set API_KEY --value "your-secret-value"

# Get a secret
dkdc secrets get API_KEY

# List all secrets
dkdc secrets list

# Export secrets
dkdc secrets export              # Shell format to stdout
dkdc secrets export -f json       # JSON format
dkdc secrets export -f dotenv     # .env format
dkdc secrets export .env          # Write to file

Archive Directories

Archive directories to the data lake:

# Archive current directory
dkdc archive

# Archive specific directory
dkdc archive /path/to/project

# Archive with custom name
dkdc archive /path/to/project --name backup-2024.zip

Configuration

Edit the configuration file:

dkdc --config

Configuration is stored at ~/.config/dkdc/config.toml.

Architecture

dkdc uses a modular architecture where each major feature is implemented as a separate Rust crate:

• dkdc-config - Configuration management
• dkdc-lake - Core DuckDB/DuckLake functionality
• dkdc-dev - Development REPL
• dkdc-files - Virtual filesystem
• dkdc-secrets - Secrets management
• dkdc-archive - Directory archiving
• dkdc-cli - Main CLI application

See ARCHITECTURE.md for detailed technical documentation.

Data Storage

All data is stored encrypted using DuckDB with the DuckLake extension:

• Location: ~/.dkdc/dkdclake/
• Metadata: SQLite database at ~/.dkdc/dkdclake/metadata.db
• Data: Encrypted files in ~/.dkdc/dkdclake/data/

Development

Setup Development Environment

# Clone the repository
gh repo clone lostmygithubaccount/dkdc
cd dkdc

# Set up the development environment
./bin/setup.sh

# Build the Rust extensions (required for first use)
uv run maturin develop

# Run code checks (lint & format)
./bin/check.sh

# Run the development REPL
uv run dkdc dev

Organization

• py/: Python source code (thin wrapper)
• rs/: Rust source code (main implementation)
• bin/: Repository utilities
• tasks/: Development tasks
• dotfiles/: Dotfiles
• websites/: Static websites

Development Workflow

1. Make changes to Rust code
2. Run uv run maturin develop to rebuild
3. Test with uv run dkdc

Building for Release

# Build wheel for distribution
uv run maturin build --release

# The wheel will be in target/wheels/
ls target/wheels/

Security

• All data is encrypted at rest using DuckLake
• Secrets are stored as encrypted BLOBs
• No secrets are logged or displayed unless explicitly requested
• Virtual environment is isolated in ~/.dkdc/venv/

Roadmap

Planned Features

• Backup/Restore - Cloud backup with triple-layer encryption
• Data Versioning - Track changes to files over time
• Access Control - User permissions and sharing
• Cloud Storage - S3/GCS/Azure backend support
• Data Lineage - Track data transformations

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

MIT License - see LICENSE for details.

Acknowledgments

Built with:

• DuckDB - In-process analytical database
• DuckLake - Encryption extension for DuckDB
• Rust - Systems programming language
• uv - Fast Python package manager