Skip to main content

Secure data pipeline for transferring sensitive data from cloud to local GPU processing

Project description

Secure Media Processor

A secure data pipeline for transferring sensitive data from cloud to local GPU processing

License: MIT Python 3.8+ Security: AES-256-GCM Tests codecov PyPI version


The Problem

You have sensitive data (medical images, confidential documents, research data) stored in the cloud. You need to process it on your local GPU workstation. How do you transfer it securely?

Traditional approaches leave data vulnerable:

  • Cloud providers can access unencrypted data
  • Data sits decrypted during processing
  • No audit trail for compliance (HIPAA, GDPR)
  • Sensitive files left on disk after processing

The Solution

┌─────────────────────────────────────────────────────────────────────────────┐
│                                                                             │
│   [Hospital/Cloud]  ══════════════►  [Your GPU Workstation]                │
│                           │                                                 │
│                     ┌─────┴─────┐                                           │
│                     │ ENCRYPTED │                                           │
│                     │  SECURE   │                                           │
│                     │ PIPELINE  │                                           │
│                     └───────────┘                                           │
│                                                                             │
│   • Data encrypted BEFORE leaving source                                   │
│   • Keys NEVER leave your workstation                                      │
│   • Decryption ONLY on your local machine                                  │
│   • Audit trail for compliance                                             │
│   • Secure deletion after processing                                       │
│                                                                             │
└─────────────────────────────────────────────────────────────────────────────┘

Secure Media Processor is a secure pipeline that ensures sensitive data is protected at every stage—from cloud storage to local GPU processing and cleanup.

Key Features

End-to-End Encryption

  • AES-256-GCM authenticated encryption
  • ECDH key exchange for multi-party transfers
  • Zero-knowledge mode — cloud provider never sees plaintext

Multi-Cloud Support

  • AWS S3
  • Google Drive
  • Dropbox
  • Azure Blob Storage (coming soon)

Compliance Ready

  • HIPAA/GDPR compliant audit logging
  • Immutable audit trail with cryptographic verification
  • Configurable retention policies

Secure by Default

  • Multi-pass secure deletion (DoD 5220.22-M)
  • Encrypted key storage
  • Checksum verification on all transfers

Quick Start

Installation

pip install secure-media-processor

Basic Usage

from secure_media_processor import Pipeline, TransferMode

# Initialize secure pipeline
pipeline = Pipeline(
    encryption_key="~/.smp/keys/master.key",
    audit_log="~/.smp/audit/"
)

# Add your cloud source
pipeline.add_source('hospital', S3Connector(
    bucket_name='patient-scans',
    region='us-east-1'
))

# Secure download to local GPU workstation
manifest = pipeline.secure_download(
    source='hospital',
    remote_path='mri-scans/patient-001/',
    local_path='/secure/gpu-workspace/',
    mode=TransferMode.ZERO_KNOWLEDGE  # Maximum security
)

# Verify data integrity
assert pipeline.verify_integrity(manifest)

# Process locally on your GPU (your code here)
results = your_ml_model.process(manifest.destination)

# Secure cleanup when done
pipeline.secure_delete(manifest.destination)

Who Is This For?

  • Medical researchers processing patient MRI/CT scans
  • Healthcare organizations meeting HIPAA requirements
  • Research institutions handling sensitive data
  • Anyone who needs to securely move data to GPU for processing

Architecture

secure-media-processor/
├── src/
│   ├── core/                    # Core pipeline functionality
│   │   ├── secure_transfer.py   # Main transfer pipeline
│   │   ├── encryption.py        # AES-256-GCM encryption
│   │   ├── audit_logger.py      # Compliance logging
│   │   └── key_exchange.py      # ECDH key exchange
│   ├── connectors/              # Cloud storage connectors
│   │   ├── s3_connector.py
│   │   ├── google_drive_connector.py
│   │   └── dropbox_connector.py
│   └── cli.py                   # Command-line interface
├── plugins/                     # Optional processing plugins
│   └── smp_medical/             # Medical imaging plugin
└── docs/                        # Documentation
    ├── architecture/
    ├── api/
    └── examples/

Plugin Architecture

The core package focuses on secure data transfer. Domain-specific processing is handled by optional plugins:

# Medical imaging (DICOM, U-Net segmentation)
pip install secure-media-processor[medical]

# Video processing (coming soon)
pip install secure-media-processor[video]

Plugins process data locally after it has been securely transferred.

Documentation

Security Model

Stage Protection
At rest (cloud) AES-256-GCM encryption
In transit AES-256-GCM + TLS
At rest (local) Encrypted with master key
Processing Decrypted only in memory
Cleanup Multi-pass secure deletion

Zero-Knowledge Transfer: In this mode, data is encrypted at the source (e.g., hospital) using a shared key derived via ECDH. The cloud provider never has access to the plaintext or decryption keys.

Compliance

HIPAA

  • Audit logging with 6-year retention
  • Access controls (local-only decryption)
  • Encryption meets requirements

GDPR

  • Data minimization support
  • Right to erasure (secure deletion)
  • Complete audit trail

CLI Usage

# Encrypt a file locally
smp encrypt sensitive-data.dcm encrypted.bin

# Download from cloud (decrypts locally)
smp download s3://bucket/path/file.enc ./local-file.dcm

# Secure delete
smp delete --secure ./sensitive-data.dcm

# View system info
smp info

Roadmap

Version 2.0 (Current Focus)

  • Core secure transfer pipeline
  • ECDH key exchange
  • HIPAA-compliant audit logging
  • Plugin architecture
  • Streaming transfer for large files
  • Azure Blob Storage connector

Future

  • Docker containerization
  • REST API server
  • Web dashboard
  • Hardware Security Module (HSM) integration

Contributing

We welcome contributions! See CONTRIBUTING.md.

Priority areas:

  • Additional cloud connectors
  • Performance optimization
  • Security auditing

Security

Security is the core mission. If you discover a vulnerability, please see our Security Policy for responsible disclosure.

License

MIT License — see LICENSE.


Secure Media ProcessorYour data, your GPU, your control.

Built for researchers who need security without compromise.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

secure_media_processor-2.1.0.tar.gz (172.4 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

secure_media_processor-2.1.0-py3-none-any.whl (141.3 kB view details)

Uploaded Python 3

File details

Details for the file secure_media_processor-2.1.0.tar.gz.

File metadata

  • Download URL: secure_media_processor-2.1.0.tar.gz
  • Upload date:
  • Size: 172.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.2.0 CPython/3.14.1

File hashes

Hashes for secure_media_processor-2.1.0.tar.gz
Algorithm Hash digest
SHA256 2f53aab0515d3df6e93afbeac090c24bac89b1392c2282683ebc36c54eeb7acd
MD5 7536bb8c16f5da3706232d5c10cbe073
BLAKE2b-256 3ea1cf0b67773f72c4e08fa47b953cf59c311d8a214f870ebab85f2cb818e578

See more details on using hashes here.

File details

Details for the file secure_media_processor-2.1.0-py3-none-any.whl.

File metadata

File hashes

Hashes for secure_media_processor-2.1.0-py3-none-any.whl
Algorithm Hash digest
SHA256 061a6c9a53f7b67ef018b52b5b444e29436ffbe3f2154bf16b20fb31960a4859
MD5 f6fb441e91435131302a7bda8a84c508
BLAKE2b-256 84d3a9bc84667d6c30adcaaefbfb13649bd189dd08beae6f0cd0b1de6c1a45c1

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page