fsspec-encrypted 0.8

Local Filesystem Example

import fsspec
from fsspec_encrypted.fs_enc_cli import generate_key

# Generate an encryption key
encryption_key = generate_key(passphrase="my_secret_passphrase", salt=b"12345432")

# Create an EncryptedFS instance (local filesystem is the default)
enc_fs = fsspec.filesystem('enc', encryption_key=encryption_key)

# Write some encrypted data to a file
enc_fs.writetext('./encfs/example.txt', 'This is some encrypted text.')

# Read the encrypted data back from the file
print(enc_fs.readtext('./encfs/example.txt'))

Pandas compatibility

Pandas uses fsspec under the hood, which lets you using the read / to methods to encrypt data Additional note, we are using the generate_key here with a passphrase and salt to allow for reusable key

import pandas as pd
from fsspec_encrypted.fs_enc_cli import generate_key

# Your encryption key
encryption_key = generate_key(passphrase="my_secret_passphrase", salt=b"12345432")

# Create a sample DataFrame
data = {
    'name': ['Alice', 'Bob', 'Charlie'],
    'age': [25, 30, 35]
}
df = pd.DataFrame(data)

# This encrypts the file to disk
df.to_csv('enc://./encfs/encrypted-file.csv', index=False, storage_options={"encryption_key": encryption_key})

print("Data written to encrypted file with key:", encryption_key.decode())

# Read and decrypt the file
df2 = pd.read_csv('enc://./encfs/encrypted-file.csv', storage_options={"encryption_key": encryption_key})

print(df2)

S3 Filesystem Example

This is an example of using encryption on top of other file systems, where we wrap S3 and encrypt or decrypt as required.

import fsspec
from cryptography.fernet import Fernet

# Generate an encryption key
encryption_key = Fernet.generate_key()

# Use the encrypted filesystem on top of an S3 filesystem
enc_fs = fsspec.filesystem('enc', encryption_key=encryption_key)

# Write some encrypted data to S3
enc_fs.writetext('s3://your-bucket/example.txt', 'This is some encrypted text.')

# Read the encrypted data back from S3
print(enc_fs.readtext('s3://your-bucket/example.txt'))

# This can also be done by wrapping the filesystem
bucket="some-bucket"
df = pd.read_csv(f'enc://s3://{bucket}/encrypted-file.csv', storage_options={"encryption_key": encryption_key})

Other Filesystems

fsspec-encrypted automatically determines the filesystem type based on the file path.

For example, if the path starts with s3://, it will use S3; otherwise, it defaults to the local filesystem. It supports any fsspec-compatible filesystem (e.g., GCS, FTP).

For wrapping the filesystem we can use enc://<other-file-system>://

Generate an Encryption Key

Store your keys appropriately - a secrets manager is an ideal solution!

# Generate a random key
# CRITICAL STORE THE KEY SOMEWHERE SECURE
key=$(fs-enc gen-key)

If you want to generate a key based on a passphrase and salt

fs-enc gen-key --passphrase 'hello world' --salt 12345432

What is a Salt?

A salt is a random 16 byte value used during the key derivation process to ensure that even if two people use the same passphrase, the derived encryption keys will be different. The salt is not a secret, but it should be unique and random for each encryption.

When encrypting data, the salt is usually stored alongside the encrypted data so that it can be used again during decryption to derive the same encryption key from the passphrase.

Encrypt data from stdin and write it to a file

# Encrypt and store locally
echo "This is sensitive data" | fs-enc encrypt --key $key --file ./encfs/encrypted-file.txt
# Decrypt
fs-enc decrypt --key $key --file ./encfs/encrypted-file.txt

Writing encrypted data to a cloud store, The following example requires the appropriate driver s3fs in this case installed and AWS env variables configured

export AWS_PROFILE=xxxxxx
pip install -U s3fs
echo "This is sensitive data" | fs-enc encrypt --key $key  --file s3://<some-bucket>/encrypted-file.txt
fs-enc decrypt --key $key --file s3://<some-bucket>/encrypted-file.txt

Setting Up for Development

1. Clone the repository:

   git clone https://github.com/thevgergroup/fsspec-encrypted.git
   cd fsspec-encrypted

2. Install the dependencies using Poetry:

   poetry install

3. After installation, any changes you make to the code will be automatically reflected when running the project.

Running Tests

The project uses pytest for testing. To run the test suite, simply use:

poetry run pytest