blob-descriptor 0.0.3

Toolkit for managing large binary files through chunking and metadata descriptors

Blob Descriptor 🗃️

A robust toolkit for managing large binary files through intelligent chunking and metadata descriptors.

☕ Support

If you find this project helpful, consider supporting me:

Features ✨

• Smart Chunking - Split large files into manageable chunks
• Metadata Tracking - Maintain comprehensive file descriptors
• Integrity Verification - MD5 checksums at file and chunk levels
• Flexible Sources - Handle local files and remote URLs
• Efficient Transfer - Parallel chunk processing
• CLI Interface - Easy command-line control

Installation 📦

pip install blob-descriptor

Or for the latest development version:

pip install git+https://github.com/jet-logic/blob_descriptor.git

Usage 🚀

Basic Workflow

1. Create descriptor and chunks:

   blob-descriptor create --cw 1M,./chunks large_file.dat
   

2. Verify integrity:

   blob-descriptor verify descriptor.bd ./chunks
   

3. Reassemble files:

   blob-descriptor assemble descriptor.bd --sink reconstructed.dat
   

Command Reference

Command	Description	Example
create	Generate descriptor and chunks	create --cs 5M bigfile.iso
verify	Check file integrity	verify desc.bd ./chunks
check	Validate chunk availability	check desc.bd --search ./backups
assemble	Reconstruct files from chunks	assemble desc.bd --sink output.img

Advanced Options

# Process only specific chunks (0-5 and 10-15)
blob-descriptor create \
  --cc "5M,./temp,upload.sh {file},0-5,10-15" \
  huge_dataset.bin

# Use custom naming pattern (mask 3)
blob-descriptor create -m 3 --cs 10M data.tar

API Example 🐍

from blob_descriptor import BlobDescriptor

# Create descriptor
bd = BlobDescriptor()
bd.add_file("large_file.bin")
desc = bd.make_descriptor(block_size=8192)

# Save to file
bd.save("descriptor.bd")

Here's a dedicated section for the naming pattern masks that you can add to your README:

Naming Pattern Masks 🏷️

Blob Descriptor provides flexible naming patterns for generated chunks through four mask options:

Available Masks

Mask	Pattern Format	Example Output
1	{md5:.5}_{total_size}_{block_size}_{index:0{block_ipad}d}_{md5:.5}	1a3f5_1048576_524288_0001_1a3f5
2	{md5:.5}_{block_size}{index:0{block_ipad}d}_{md5:.5}_{total_size}	1a3f5_512K0001_1a3f5_1048576
3	{md5:.5}_{block_size}{index:0{block_ipad}d}_{total_size}	1a3f5_512K0001_1048576
4	{md5:.5}_{block_size}{index:0{block_ipad}d}	1a3f5_512K0001

Format Variables

• {md5:.5}: First 5 chars of MD5 hash
• {total_size}: Full blob size in bytes
• {block_size}: Chunk size in bytes (mask 1) or human-readable (masks 2-4)
• {index}: Chunk sequence number
• {block_ipad}: Zero-padding width based on total chunks

Usage Examples

Set mask when creating descriptor:

blob-descriptor create -m 3 --cs 10M bigfile.iso

Programmatic configuration:

from blob_descriptor import set_mask, mask3
set_mask(mask3)  # Use mask pattern 3

Mask Comparison

# Mask 1 (Default)
1a3f5_1048576_524288_0001_1a3f5

# Mask 2 (Size suffix)
1a3f5_512K0001_1a3f5_1048576

# Mask 3 (Compact)
1a3f5_512K0001_1048576

# Mask 4 (Minimal)
1a3f5_512K0001

Choose masks based on your needs:

• Mask 1: Full technical details
• Mask 2: Balanced readability
• Mask 3: Clean with size info
• Mask 4: Minimal footprint

The pattern affects both:

• Chunk filenames
• Descriptor metadata
• Verification outputs