datasovereign-scout 0.1.0

A privacy-focused tool to detect orphaned and sensitive datasets.

🔍 DataSovereign-Scout

Developed by: Dr. T.M. Usha

LinkedIn Profile • [e-mail] • [phone]

---

A privacy-focused Python CLI tool to detect orphaned datasets and ghost data.

Features • Installation • Quick Start • Documentation • Contributing

---

📖 Table of Contents

• What is DataSovereign-Scout?
• The Problem: Ghost Data
• Features
• Installation
• Quick Start
• How It Works
• Advanced Usage
• Examples
• Architecture
• Contributing
• License
• Contact

---

🎯 What is DataSovereign-Scout?

DataSovereign-Scout is a command-line tool that helps organizations and individuals discover "ghost data" - forgotten files, orphaned backups, leaked credentials, and unauthorized datasets hiding in your infrastructure.

Think of it as a home inspector for your data. You tell it what files should be there (via a manifest), and it alerts you to everything else that shouldn't exist.

🌟 Perfect For:

• 🏢 DevOps Teams - Finding leaked .env files and credentials before deployment
• 🔐 Security Auditors - Discovering unauthorized data before compliance reviews
• 📊 Data Engineers - Identifying orphaned datasets consuming storage
• 🎓 Researchers - Ensuring sensitive research data is properly tracked
• 💼 Startups - Preventing data leaks during rapid development

---

🚨 The Problem: Ghost Data

What is Ghost Data?

Ghost Data refers to files that exist on your system but are:

• ❌ Not documented in any inventory
• ❌ Not actively used or maintained
• ❌ Potentially sensitive (backups, credentials, PII)
• ❌ Unknown to current team members
• ❌ Compliance/privacy risks

Real-World Examples:

❌ forgotten_backup_2023.csv      (Contains customer data)
❌ .env                            (Database passwords exposed)
❌ aws_production.pem              (Private keys for cloud access)
❌ database_dump_old.sql           (PII from deleted users)
❌ test_credentials.json           (API keys from former developers)

The Risk: These files can lead to:

• 🔓 Data breaches
• 💰 GDPR/CCPA fines
• 😰 Reputational damage
• 📉 Customer trust loss

---

✨ Features

🎯 Core Capabilities

• 📋 Manifest-Based Scanning - Define your "known good" data in a simple YAML file
• 🔍 Smart Detection - Identifies files by extension AND risky keywords
• 🔐 Security-Focused - Flags .pem, .key, .env, .p12, and other sensitive formats
• 📊 Rich Terminal Output - Beautiful, color-coded results powered by rich
• ⚡ Fast & Lightweight - Scans thousands of files in seconds
• 🧠 Intelligent Filtering - Auto-skips .git, venv, __pycache__, etc.

🕵️ Detection Rules

Extension-Based:

• Data Files: .csv, .json, .sql, .db, .sqlite, .parquet, .xlsx
• Secrets: .pem, .key, .env, .p12, .kdbx
• Logs: .log

Keyword-Based:

• Filenames containing: backup, dump, secret, password, token, credential

---

📦 Installation

Prerequisites

• Python 3.8 or higher
• pip (Python package manager)

Install from PyPI

pip install datasovereign-scout

Install from Source

git clone https://github.com/DrTMUSHACoder/py-datasovereign-scout.git
cd py-datasovereign-scout
pip install -e .

Verify Installation

ds-scout --help

You should see:

Usage: ds-scout [OPTIONS] COMMAND [ARGS]...

  DataSovereign-Scout: Detect Ghost Data and Orphaned Datasets.

---

🚀 Quick Start

Step 1: Create a Manifest

Create a file named manifest.yaml listing your approved data files:

assets:
  - data/users.csv
  - reports/quarterly_2024.xlsx
  - configs/app_settings.json

Step 2: Run a Scan

ds-scout scan --manifest manifest.yaml --path ./

Step 3: Review Results

Expected Output:

Scouting for Ghost Data...
Manifest: manifest.yaml
Scan Path: ./

╔═══════════════════════════════════════════════════════════╗
║ CRITICAL: Found 3 orphaned datasets!                      ║
╚═══════════════════════════════════════════════════════════╝

Orphaned Assets (Ghost Data)
╔═══════════════════════════════════════════════════════════╗
║ Path                                                      ║
╠═══════════════════════════════════════════════════════════╣
║ /project/data/forgotten_backup_2023.csv                   ║
║ /project/.env                                             ║
║ /project/secrets/aws_production.pem                       ║
╚═══════════════════════════════════════════════════════════╝

Total Scanned Files: 15

---

🔧 How It Works

The Workflow

graph LR
    A[Create manifest.yaml] --> B[Run ds-scout scan]
    B --> C[Scanner reads manifest]
    C --> D[Walks filesystem]
    D --> E{File matches<br/>extensions or<br/>keywords?}
    E -->|Yes| F{File in<br/>manifest?}
    E -->|No| G[Ignore]
    F -->|Yes| H[Mark as Safe]
    F -->|No| I[FLAG as Ghost Data]
    I --> J[Report to User]

Detection Logic

# Pseudo-code
for file in filesystem:
    if file.extension in SENSITIVE_EXTENSIONS:
        if file not in manifest:
            FLAG_AS_GHOST_DATA(file)
    
    if any(keyword in file.name for keyword in RISKY_KEYWORDS):
        if file not in manifest:
            FLAG_AS_GHOST_DATA(file)

---

🎓 Advanced Usage

Scan a Specific Directory

ds-scout scan --manifest manifest.yaml --path ./production_data

Include Hidden Files

By default, hidden directories (.git, .venv) are skipped. To include them, modify the scanner source code or open an issue for this feature.

Custom Extensions (Future Feature)

# Coming soon
ds-scout scan --manifest manifest.yaml --path ./ --extensions .dat,.bin

---

📚 Examples

Example 1: DevOps Pre-Deployment Check

Scenario: You're about to deploy a Docker container. Check for leaked credentials.

# Create manifest
cat > manifest.yaml << EOF
assets:
  - src/config.json
  - data/sample.csv
EOF

# Scan
ds-scout scan --manifest manifest.yaml --path ./

# Result: Finds forgotten .env file with database password

---

Example 2: GDPR Compliance Audit

Scenario: Verify no unauthorized customer data exists before a compliance review.

# manifest.yaml
assets:
  - database/customers_2024.db
  - exports/approved_report.xlsx

ds-scout scan --manifest manifest.yaml --path ./

# Finds: old_customer_backup.csv (not in manifest)
# Action: Securely delete or add to consent records

---

Example 3: Research Data Management

Scenario: Ensure all research datasets are properly documented.

# manifest.yaml
assets:
  - experiments/trial_001.csv
  - experiments/trial_002.csv

ds-scout scan --manifest manifest.yaml --path ./experiments

# Finds: pilot_study_draft.csv (orphaned from previous researcher)

---

🏗️ Architecture

Project Structure

datasovereign-scout/
├── src/
│   └── datasovereign_scout/
│       ├── __init__.py        # Package initializer
│       ├── cli.py             # Command-line interface
│       ├── manager.py         # Orchestrates scan logic
│       ├── manifest.py        # Manifest parser
│       ├── scanners.py        # Filesystem scanner
│       └── detector.py        # Detection rules
├── tests/                     # Unit tests
├── pyproject.toml             # Package configuration
├── README.md                  # This file
└── LICENSE                    # MIT License

Key Components

Component	Responsibility
cli.py	Argument parsing, user interaction
manager.py	Coordinates manifest + scanner
scanners.py	Walks filesystem, applies rules
manifest.py	Parses YAML manifest
detector.py	Detection algorithms

---

🤝 Contributing

We welcome contributions! Here's how you can help:

Ways to Contribute

• 🐛 Report bugs via GitHub Issues
• 💡 Suggest features
• 📝 Improve documentation
• 🧪 Write tests
• 🔧 Submit pull requests

Development Setup

# Clone the repo
git clone https://github.com/DrTMUSHACoder/py-datasovereign-scout.git
cd py-datasovereign-scout

# Install in editable mode
pip install -e .

# Run tests (when added)
pytest tests/

Coding Standards

• Follow PEP 8
• Add type hints
• Write docstrings
• Include tests for new features

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

MIT License

Copyright (c) 2026 Dr. T.M. Usha

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction...

---

📞 Questions or Feedback?

Feel free to reach out via the contact information at the top of this README or open an issue on GitHub!

---

🌟 Acknowledgments

• Built with Python 3.8+
• Powered by Rich for terminal output
• Inspired by real-world data governance challenges

---

🔗 Related Projects

• detect-secrets - Pre-commit hook for secret detection
• truffleHog - Searches for secrets in Git history
• GitGuardian - Automated secret scanning

---

📊 Project Stats

---

Made with ❤️ for data privacy and security

If this tool helped you, please ⭐ star the repository!