Why s3manifesto?

In big data and ETL pipelines, efficiently managing thousands or millions of files becomes a critical bottleneck. s3manifesto solves this by providing:

• Metadata Organization: Consolidate file metadata (URI, size, record count, ETag) into easily manageable collections

• Intelligent Partitioning: Automatically group files into balanced batches for optimal parallel processing

• Divide-and-Conquer Optimization: Implement efficient distributed processing workflows with predictable resource utilization

Instead of dealing with individual file metadata scattered across your data lake, s3manifesto enables you to treat collections of files as single, manageable units with powerful partitioning capabilities.

Core Concepts

1. Manifest as Metadata Collection

A manifest represents metadata for a collection of data files, where each data file contains:

• S3 URI: File location identifier

• ETag: Data integrity verification hash

• Size: File size in bytes for resource planning

• Record Count: Number of records for workload estimation

• Additional attributes: Extensible metadata as needed

2. Two-File Storage System

Each manifest consists of two files stored in S3:

• Manifest Summary File (JSON): Aggregate statistics and references

• Manifest Data File (Parquet): Detailed per-file metadata in parquet format

This design enables quick access to summary information without loading detailed metadata, optimizing both storage and retrieval performance.

3. Intelligent File Partitioning

Manifest files can partition large collections into balanced groups using the Best Fit Decreasing (BFD) algorithm:

• By Total Size: Group files into batches of ~100MB each for memory optimization

• By Record Count: Group files into batches of ~10M records each for processing time consistency

• Optimal Distribution: Ensures balanced workloads across parallel workers

• Divide-and-Conquer Ready: Perfect for distributed processing frameworks

Example: Transform 10,000 files into 50 balanced groups of ~200 files each, with each group totaling approximately your target size or record count.

Quick Example

from s3manifesto import ManifestFile, DataFile

# Create manifest from file metadata
data_files = [
    DataFile(uri="s3://bucket/file1.json", size=1000000, n_record=1000, etag="abc123"),
    DataFile(uri="s3://bucket/file2.json", size=2000000, n_record=2000, etag="def456"),
    DataFile(uri="s3://bucket/file3.json", size=3000000, n_record=3000, etag="ghi789")
]

manifest = ManifestFile.new(
    uri="s3://bucket/manifest-data.parquet",
    uri_summary="s3://bucket/manifest-summary.json",
    data_file_list=data_files
)

# Write to S3
manifest.write(s3_client)

# Read from S3
manifest = ManifestFile.read("s3://bucket/manifest-summary.json", s3_client)

# Partition files for parallel processing
groups = manifest.partition_files_by_size(target_size=100_000_000)  # 100MB groups
groups = manifest.partition_files_by_n_record(target_n_record=10_000_000)  # 10M record groups

Install

s3manifesto is released on PyPI, so all you need is to:

$ pip install s3manifesto

To upgrade to latest version:

$ pip install --upgrade s3manifesto