Skip to main content

A package that is able to constantly write encrypted bytes to disk

Project description

GnuPG Eternal - Large File Encryption with Line Modification

GnuPG Eternal enables developers and data scientists to encrypt and decrypt large files while modifying them in memory. It is specifically designed for de-tokenization + encryption of sensitive data such as account numbers and social insurance numbers, where raw data is not allowed to live on the local storage of the system.

Use Case

One primary example is for transferring and processing PANs. Provided is a file of customers' PANs that are tokenized. The requirement is to send the encrypted and de-tokenized PANs to the receiver.

However, because PANs are highly sensitive, the de-tokenized/raw PANs cannot touch the local storage of the system. Hence, de-tokenization and encryption need to happen in system memory.

Advantage

Traditionally, the content of the whole file can be loaded into memory if wanted, and the developer/data scientist can manipulate the data depending on his/her needs.

As the file size increases to a certain point where the system cannot hold it any more, the program would crash.

GnuPG Eternal allows the developer/data scientist to manipulate and encrypt files regardless of their sizes: that is, the maximum space consumption is constant, provided that the file is not one massive line of data.

Quick Start

Install via pip:

pip install gpgeternal

Basic Example

With the file to manipulate on the local storage, define the path to the file and the path to the processed/manipulated file. The file to process must exist.

Define the recipient of the encrypted file and the manipulation method for each line (ex. de-tokenization method that returns the de-tokenized line).

import gpgeternal

def manipulation_func(line):
    # The example manipulation method
    line = line.replace('Foobar', 'cipher')
    return line

def main():
    file_to_encrypt = 'test.txt'  # File to manipulate and encrypt
    recipient = 'yiren.zhou@edu.uwaterloo.ca'  # Email of the recipient (GnuPG)
    output_file = 'test.pgp'   # Name of the encrypted & manipulated file

    gpgeternal.encrypt_large_file(recipient, file_to_encrypt, output_file, manipulation_func)

if __name__ == '__main__':
    main()

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for gpgeternal, version 0.9
Filename, size File type Python version Upload date Hashes
Filename, size gpgeternal-0.9.tar.gz (3.1 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page