pgpack 0.3.0.4

PGCopy dump packed into LZ4, ZSTD or uncompressed with meta data information packed into zlib.

PGPack format

Storage format for PGCopy dump packed into LZ4, ZSTD or uncompressed with meta data information packed into zlib

PGPack structure

• header b"PGPACK\n\x00" 8 bytes
• unsigned long integer zlib.crc32 for packed metadata 4 bytes
• unsigned long integer zlib packed metadata length 4 bytes
• zlib packed metadata
• unsigned char compression method 1 byte
• unsigned long long integer packed pgcopy data length 8 bytes
• unsigned long long integer unpacked pgcopy data length 8 bytes
• packed pgcopy data

Installation

From pip

pip install pgpack

From local directory

pip install .

From git

pip install git+https://github.com/0xMihalich/pgpack

Metadata format

Metadata for PGCopy dump contained Column names and OID Types

Decompressed metadata structure

list[
    list[
        column number int,
        list[
            column name str,
            column oid int,
            column lengths int,
            column scale int,
            column nested int,
        ]
    ]
]

Compression methods

• NONE (value = 0x02) PGCopy dump without compression
• LZ4 (value = 0x82) PGCopy dump with lz4 compression
• ZSTD (value = 0x90) PGCopy dump with zstd compression

Get ENUM for set compression method

from pgpack import CompressionMethod

compression_method = CompressionMethod.NONE  # no compression
compression_method = CompressionMethod.LZ4  # lz4 compression
compression_method = CompressionMethod.ZSTD  # zstd compression (default)

Class PGPackReader

Initialization parameters

• fileobj - BufferedReader object (file, BytesIO e t.c)

Methods and attributes

• metadata - metadata in bytes
• columns - List columns names
• pgtypes - List PGOid for all columns
• pgparam - List PGParam for all columns
• pgcopy_compressed_length - integer packed pgcopy data length
• pgcopy_data_length - integer unpacked pgcopy data length
• compression_method - CompressionMethod object
• compression_stream - BufferedReader object for decompress data
• pgcopy_start - integer offset for start pgcopy compressed data
• pgcopy - PGCopyReader object
• to_rows() - Method for reading uncompressed PGCopy data as generator python objects
• to_pandas() - Method for reading uncompressed PGCopy data as pandas.DataFrame
• to_polars() - Method for reading uncompressed PGCopy data as polars.DataFrame
• to_bytes() - Method for reading uncompressed PGCopy data as generator bytes

Class PGPackWriter

Initialization parameters

• fileobj - BufferedWriter object (file, BytesIO e t.c)
• metadata - metadata in bytes (default is None)
• compression_method - CompressionMethod object (default is CompressionMethod.ZSTD)

Methods and attributes

• columns - List columns names
• pgtypes - List PGOid for all columns
• pgparam - List PGParam for all columns
• pgcopy_compressed_length - integer packed pgcopy data length set to 0 as initialized
• pgcopy_data_length - integer unpacked pgcopy data length set to -1 as initialized
• pgcopy_start - integer offset for start pgcopy compressed data set to current offset as initialized
• pgcopy - PGCopyWriter object
• from_rows(dtype_data) - Write PGPack file from python objects. Parameter: dtype_data as python iterable object
• from_pandas(data_frame) - Write PGPack file from pandas.DataFrame. Parameter: data_frame as pandas.DataFrame
• from_polars(data_frame) - Write PGPack file from polars.DataFrame. Parameter: data_frame as polars.DataFrame
• from_bytes(bytes_data) - Write PGPack file from bytes. Parameter: bytes_data as bytes iterable object

Errors

• PGPackError - Base PGPack error
• PGPackHeaderError - Error header signature
• PGPackMetadataCrcError - Error metadata crc32
• PGPackModeError - Error fileobject mode