fernet-files 0.1.0

Encryption/decryption of files with cryptography.fernet, but with file-like methods

Fernet Files - encryption/decryption of files with cryptography.fernet, but with file-like methods

Fernet encryption requires all data to be encrypted or decrypted at once. This is memory intensive, and it is slow if you only want to read part of a file. Fernet Files provides a simple interface that breaks data up into chunks when encrypting and decrypting them to lower memory usage. Data is only encrypted when it's necessary to do so.

You may treat the class similar to a file: it has read, write, seek and close methods. It can also be context managed, so you can close it using a with statement.

Contents

• Example usage
• Requirements
• Installation
• Benchmarks
• Documentation for module users
• Documentation for module developers

Example usage

from fernet_files import FernetFile
key = FernetFile.generate_key() # Keep this
with FernetFile(key, "filename.bin") as f:
  # Use like a normal file
  f.write(b'123456789') # Returns 9
  f.seek(4) # Returns 4
  f.read(3) # Returns b'567'
  ...

# If you check the file after closing (leaving the with statement)
# the contents will be encrypted and unreadable without using the module

# If you use the same key, you can then read the data again
with FernetFile(key, "filename.bin") as f:
    f.read() # Returns b'123456789'

Requirements

• cryptography >= 41.0.4 (might break in the future, I'll try and keep an eye on it)
• Python 3.10 or greater (3.10, 3.11 and 3.12 tested)

custom_fernet.py is based on cryptography 41.0.4. If the file it's based on is updated, it might break this module. If this has happened, created an issue in this repository.

Installation

Pending upload to pip

Benchmarks

Info

Based on these benchmarks, the default chunk size is now 64KiB. It is recommended you change this to a smaller value (e.g. 4KiB) for smaller files.

• Tests can be found in benchmark.py and raw results in benchmark_results.py
• All values are given to 3sf (significant figures).
• All times are given in milliseconds.
• All bytes are given with their units.

Time to encrypt

Data

Raw data size	1B	16B	256B	4KiB	64KiB	1MiB	16MiB	256MiB	4GiB
Fernet	127ms	0.703ms	0.802ms	2.16ms	2.79ms	15.8ms	240ms	5790ms	106000ms
FernetNoBase64	1.77ms	2.00ms	1.76ms	1.89ms	2.59ms	8.01ms	127ms	3560ms	73500ms
FF 16B	1.72ms	2.17ms	8.66ms	46.1ms	368ms	5530ms	260000ms	NT	NT
FF 256B	2.31ms	2.19ms	2.17ms	7.97ms	32.2ms	338ms	21200ms	NT	NT
FF 4KiB	3.20ms	2.82ms	2.79ms	2.92ms	9.23ms	34.2ms	1670ms	26600ms	400000ms
FF 64KiB	3.85ms	4.14ms	4.12ms	4.32ms	4.42ms	17.2ms	214ms	3020ms	49400ms
FF 1MiB	16.0ms	22.1ms	14.8ms	15.2ms	17.5ms	13.5ms	174ms	2590ms	39600ms
FF 16MiB	161ms	157ms	158ms	157ms	165ms	182ms	159ms	2400ms	37300ms

Observations

• Fernet encryption for 1 byte is an outliar, likely influenced by external factors
• FernetNoBase64 is slower than Fernet for data less than 4KiB, and faster for larger data.
• When using FernetFiles, encryption with low chunk sizes is slow.
• As chunk size increases, encryption speed increases, as chunk size approaches the size of the raw data.
• This gives diminishing returns, as can be seen in the 4GiB tests: 16MiB chunks are only marginally faster than 1MiB chunks.
• When chunk size is equal to raw data size, this is when encryption is fastest.
• As chunk size increases past the data size, encryption slows down again. For any given chunk size, the new encryption speed seems to be relatively constant (if data size is less than the chunk size).
• 64KiB appears to be the optimal default chunk size that covers the most reasonable file sizes. 4KiB may be more reasonable for smaller files.

Time to read first byte (decryption)

Data

Raw data size	1B	16B	256B	256MiB	4GiB
Fernet	202ms	161ms	...	3570ms	54100ms
FernetNoBase64	16.0ms	27.6ms	...	2010ms	30700ms
FF 16B	18.1ms	33.0ms	...	NT	NT
FF 256B	35.0ms	251ms	...	NT	NT
FF 4KiB	309ms	22.4ms	...	54.4ms	84.6ms
FF 64KiB	36.0ms	38.1ms	...	287ms	43.5ms
FF 1MiB	94.7ms	255ms	...	90.7ms	58.1ms
FF 16MiB	548ms	608ms	...	411ms	377ms

Observations

• FernetNoBase64 is much faster at Fernet at decryption. This is more noticeable the smaller the input data.
• Read times for FernetFiles are too inconsistent for small chunk sizes to draw conclusions. This is likely due to inconsistency in the time it takes to read a file. For this reason, some data has been omitted due to not providing useful insight. Perhaps in future, a random read test, or manipulating data in memory, would be more useful.
• Very large chunk sizes take longer to read, but this isn't noticeable for most typical chunk sizes (less than 1MiB).
• Regardless of the inconsistency, reading a byte of a FernetFile is almost always faster than with Fernet (Fernet requires you to decrypt the entire file).

Peak memory usage

Data

Raw data size	1B	16B	256B	4KiB	64KiB	1MiB	16MiB	256MiB	4GiB
Fernet	10.7KiB	9.69KiB	11.1KiB	36.1KiB	436KiB	6.68MiB	107MiB	1.67GiB	26.67GiB
FernetNoBase64	10.6KiB	9.69KiB	10.3KiB	25.4KiB	265KiB	4.01MiB	64.0MiB	1.00GiB	16.0GiB
FF 16B	11.2KiB	10.1KiB	11.8KiB	10.9KiB	11.2KiB	11.3KiB	11.0KiB	NT	NT
FF 256B	11.7KiB	11.2KiB	11.1KiB	12.7KiB	11.8KiB	12.0KiB	12.0KiB	NT	NT
FF 4KiB	30.2KiB	29.9KiB	30.1KiB	29.9KiB	30.5KiB	30.7KiB	30.8KiB	30.9KiB	30.4KiB
FF 64KiB	330KiB	330KiB	330KiB	334KiB	330KiB	331KiB	331KiB	331KiB	331KiB
FF 1MiB	5.01MiB	5.01MiB	5.01MiB	5.01MiB	5.07MiB	5.01MiB	5.01MiB	5.01MiB	5.01MiB
FF 16MiB	80.0MiB	80.0MiB	80.0MiB	80.0MiB	80.1MiB	81.0MiB	80.0MiB	80.0MiB	80.0MiB

Observations

• Fernet uses approximately 10KB $+$ datasize $\times$ 6.67
• FernetNoBase64 uses approximately 10KB $+$ datasize $\times$ 4
• As a result, FernetFiles uses approximately 10KB $+$ chunksize $\times$ 5, though this can vary slightly due to overhead from padding. (the additional x1 is the memory used to store the chunk)
• FernetFiles offers a trade off: much less memory usage, in exchange for increased processing time. If your objective is to encrypt an extremely large file as quickly as possible, then set the chunk size equal to your available memory divided by 6.
• Memory usage during decryption has not been included in a table because it is very similar to the table. Not this does not mean that the memory usage is similar, but that the trends seen in the data are similar.

Documentation for module users

class fernet_files.FernetFile(self, key, file, chunksize=65536)

Parameters:

• key - A key (recommended) or a fernet_files.custom_fernet.FernetNoBase64 object
• • A key must be 32 random bytes. Get using fernet_files.FernetFile.generate_key() and store somewhere secure
• • Alternatively, pass in a fernet_files.custom_fernet.FernetNoBase64 object
• file - Accepts a filename as a string, or a file-like object. If passing in a file-like object, it would be opened in binary mode.
• chunksize - The size of chunks in bytes.
• • Bigger chunks use more memory and take longer to read or write, but smaller chunks can be very slow when trying to read/write in large quantities.
• • Bigger chunks apply padding so a very large chunksize will create a large file. Every chunk has its own metadata so a very small chunk size will create a large file.
• • Defaults to 64KiB (65536 bytes).

method fernet_files.FernetFile.read(self, size=-1)

Reads the number of bytes specified and returns them.

Parameters:

• size - Positive integer. If -1 or not specified then read to the end of the file.

method fernet_files.FernetFile.write(self, b)

Writes the given bytes. Returns the number of bytes written.

Parameters:

• b - The bytes to be written.

method fernet_files.FernetFile.seek(self, offset, whence=os.SEEK_SET)

Can be called as:

• seek(self, offset, whence)
• seek(self, offset, whence=whence)

Moves through the file by the specified number of bytes. "whence" determines what this is relative to. Returns your new absolute position as an integer.

Parameters:

• offset - Integer. Move this number of bytes relative to whence.
• whence - Ignored if using a BytesIO object. Accepted values are:
• • os.SEEK_SET or 0 - relative to the start of the stream
• • os.SEEK_CUR or 1 - relative to the current stream position
• • os.SEEK_END or 2 - relative to the end of the stream (use negative offset)

method fernet_files.FernetFile.close(self)

Writes all outstanding data closes the file. Returns None unless the file is a BytesIO object, in which case it returns the object without closing it.

static method fernet_files.FernetFile.generate_key()

Static method used to generate a key. Acts as a pointer to custom_fernet.FernetNoBase64.generate_key().

bool fernet_files.FernetFile.closed

Boolean attribute representing whether the file is closed or not. True means the file is closed, False means the file is open. It is highly recommended that you do not modify this, and use the close method instead.

bool fernet_files.FernetFile.writeable

Boolean attribute representing whether the file can be written to or not. True if you can write to the file, False if you can't. Will only be False if you passed in a read-only file. It is highly recommended that you do not modify this.

Misc

int fernet_files.META_SIZE

It is highly recommended you don't modify this. Defaults to 8. META_SIZE represented as $M$ in formulae.

The size of a file's metadata in bytes is $2M$. The first number is a little-endian unsigned $(8M)$-bit integer, representing how many chunks are in the file. The second number is a little-endian unsigned $(8M)$-bit integer, representing the size of the last chunk's padding.

This simultaneously limits both chunksize and the number of chunks a file can have:

• A chunk can have a max size of $2^{8M}-1$ bytes (default 18,446,744,073,709,551,615)
• A file can have a max $2^{8M}-1$ chunks (default 18,446,744,073,709,551,615)

You can change this value in order to bypass these limitations for future-proofing, however, the value you use must be consistent when reading and writing to the same file. Therefore, I recommend you don't change it unless you absolutely have to, for compatibility reasons.

int fernet_files.DEFAULT_CHUNKSIZE

The chunksize that is used by default, currently 4096 bytes.

class fernet_files.custom_fernet.FernetNoBase64(self, key)

cryptography.fernet.Fernet without any base64 encoding or decoding. See custom_fernet.py for more info.

Documentation for module developers

class fernet_files.FernetFile

(RawIOBase or BufferedIOBase or BytesIO) fernet_files.FernetFile.__file

The file object used for reading and writing. If a filename is provided then this is opened in "wb+" mode.

int fernet_files.FernetFile.__last_chunk

The chunk number of the last chunk in the file. Chunks are numbered sequentially, starting from 0.

int fernet_files.FernetFile.__last_chunk_padding

The last chunk is padded with null bytes to fill the size of the chunk. This integer stores the size of the padding in bytes.

int fernet_files.FernetFile.__data_chunksize

The amount of data in a chunk in bytes.

int fernet_files.FernetFile.__chunksize

The size in bytes that chunks take up once they're written to disk. This is calculated with the following formula, where c is chunksize:

True chunksize = $c + 73 - (c \mod{16})$

This formula calculates the size of a Fernet token, based on the Fernet specification.

bool fernet_files.FernetFile.__chunk_modified

Boolean attribute representing whether the data stored in self.__chunk has been modified relative to the data stored within the self.__file. True if the chunk has been modified, False if it hasn't.

property int fernet_files.FernetFile._pos_pointer

Stores the Fernet file's current position in the chunk in bytes. The getter returns self.__pos_pointer. The setter ensures that $0\leq$ _pos_pointer $<$ chunksize. If it isn't, then it wraps the value round by adding or subtracting the chunksize, modifying the chunk pointer to compensate.

int fernet_files.FernetFile.__pos_pointer

Stores the value for self._pos_pointer.

property int fernet_files.FernetFile._chunk_pointer

Stores the Fernet file's current chunk number. The getter returns self.__chunk_pointer. The setter modifies this value. Before it switching chunks it checks if the current chunk has been modified and writes it if it has. After switching chunks, we read the new chunk into memory.

int fernet_files.FernetFile.__chunk_pointer

Stores the value for self._chunk_pointer.

method fernet_files.FernetFile.__goto_current_chunk(self)

Moves our position in self.__file to the location represented by the chunk pointer, taking into account the metadata at the start of the file. Calculated as follows: take the number of the chunk you're currently on, multiply by the size of chunks when they're written to disk. Take the META_SIZE, multiply that by 2 and add it to the number you had before.

method fernet_files.FernetFile.__get__file_size(self)

Calculate the size of the data contained within the file in bytes using the file's metadata. This is the size of the data, not the size of what is written to disk. Calculated as follows: take the number of the last chunk and add 1 to get the total number of chunks (because counting starts at 0). Multiply this by the chunksize. Finally, subtract the size of the padding used on the last chunk.

method fernet_files.FernetFile.__read_chunk(self)

Reads and decrypts the current chunk, turns it into a BytesIO object, stores that object in self.__chunk and returns it. If the chunk has been modified, it is already loaded into memory so no file operations are done. Also responsible for removing padding if the chunk being read is the last chunk.

method fernet_files.FernetFile.__write_chunk(self)

Encrypts and writes the chunk, and sets self.__chunk_modified to False. Also responsible for applying padding and modifying the metadata at the start of the file if this is the last chunk.

method fernet_files.FernetFile.__enter__(self)

Returns self to allow context management.

method fernet_files.FernetFile.__exit__(self, exc_type, exc_value, exc_traceback)

Calls self.close and returns None.

method fernet_files.FernetFile.__del__(self)

Calls self.close and returns None.

custom_fernet.FernetNoBase64 fernet_files.FernetFile.__fernet

FernetNoBase64 object created from the key provided. Used for encryption and decryption.