Skip to main content

pfc-migrate — Convert existing JSONL archives to PFC format

You already have logs stored somewhere — gzip, zstd, bzip2, or lz4 — on disk, on S3, in Glacier, on Azure, or on GCS. Switching to PFC-JSONL means converting them once. This tool does that, wherever your files live.

PyPI License: MIT Python PFC-JSONL


Storage Backends

Backend Status Command
Local filesystem ✅ v0.1.0 pfc-migrate convert
Amazon S3 / S3 Glacier ✅ v0.2.0 pfc-migrate s3 / pfc-migrate glacier
Azure Blob Storage ✅ v0.3.0 pfc-migrate azure
Google Cloud Storage ✅ v0.3.0 pfc-migrate gcs

Why convert?

Once your archives are in PFC format, DuckDB can query them directly — without decompressing the whole file first:

INSTALL pfc FROM community;
LOAD pfc;
LOAD json;

-- Query just one hour from a 30-day archive
SELECT line->>'$.level' AS level, line->>'$.message' AS message
FROM read_pfc_jsonl(
    '/var/log/pfc/app_2026-03-01.pfc',
    ts_from = epoch(TIMESTAMPTZ '2026-03-01 14:00:00+00'),
    ts_to   = epoch(TIMESTAMPTZ '2026-03-01 15:00:00+00')
);
Tool 1h query on 30-day archive Storage savings vs gzip
gzip Decompress full 30-day file —
zstd Decompress full 30-day file —
PFC-JSONL Decompress ~1/720 of the file 25% smaller than gzip

~9% compression ratio on typical JSONL log data (25% smaller than gzip, 37% smaller than zstd).


Zero egress cost

Cloud conversions run in-region: download → convert → upload, without ever routing through your laptop or billing you for egress. For Glacier, paying the one-time retrieval cost to convert is worth it — smaller PFC files mean lower future storage, retrieval, and query costs forever.


Input Formats

Format Extension Extra dependency
gzip .jsonl.gz stdlib ✅
bzip2 .jsonl.bz2 stdlib ✅
zstd .jsonl.zst pip install pfc-migrate[zstd]
lz4 .jsonl.lz4 pip install pfc-migrate[lz4]
Plain JSONL .jsonl stdlib ✅

Requirements

The pfc_jsonl binary must be installed on the machine doing the conversion:

curl -L https://github.com/ImpossibleForge/pfc-jsonl/releases/latest/download/pfc_jsonl-linux-x64 \
     -o /usr/local/bin/pfc_jsonl && chmod +x /usr/local/bin/pfc_jsonl

Platform: Linux x86_64. macOS binary coming soon.


Install

pip install pfc-migrate

# With zstd support
pip install pfc-migrate[zstd]

# With S3/Glacier support
pip install pfc-migrate[s3]

# With Azure Blob Storage support
pip install pfc-migrate[azure]

# With Google Cloud Storage support
pip install pfc-migrate[gcs]

# Everything
pip install pfc-migrate[all]

Usage — Local filesystem

# Single file (output auto-named: logs.pfc + logs.pfc.bidx)
pfc-migrate convert logs.jsonl.gz

# Explicit output
pfc-migrate convert logs.jsonl.gz logs.pfc

# Entire directory
pfc-migrate convert --dir /var/log/archive/ --output-dir /var/log/pfc/

# Recursive + verbose
pfc-migrate convert --dir /mnt/logs/ -r -v

# Delete originals after successful conversion
pfc-migrate convert --dir /var/log/archive/ --output-dir /var/log/pfc/ --delete

Verbose output:

Found 12 file(s) to convert

  → app_2026-03-01.jsonl.gz  [gz]
     145.3 MB → 13.1 MB  (9.0% of original)  ✓ app_2026-03-01.pfc
  → app_2026-03-02.jsonl.gz  [gz]
     138.7 MB → 12.4 MB  (8.9% of original)  ✓ app_2026-03-02.pfc
  ...

Done: 12 converted, 0 failed
  Input  : 1721.4 MB (decompressed)
  Output : 154.9 MB
  Ratio  : 9.0%

Usage — Amazon S3 / S3 Glacier

Conversion happens in-region (download to temp dir → convert → upload). No egress charges.

# Single object
pfc-migrate s3 \
  --bucket my-logs \
  --key archive/app_2026-03.jsonl.gz \
  --out-bucket my-logs-pfc \
  --out-prefix converted/

# All objects matching a prefix
pfc-migrate s3 \
  --bucket my-logs \
  --prefix archive/2026-03/ \
  --out-bucket my-logs-pfc \
  --out-prefix converted/2026-03/ \
  --format gz \
  --verbose

# With explicit credentials
pfc-migrate s3 \
  --bucket my-logs --prefix archive/ \
  --out-bucket my-logs-pfc --out-prefix pfc/ \
  --access-key AKIA... --secret-key ... --region us-east-1

# Glacier (Expedited retrieval, converts and re-archives)
pfc-migrate glacier \
  --bucket my-glacier-logs \
  --prefix 2025/ \
  --out-bucket my-glacier-pfc \
  --retrieval-tier Expedited

Credentials are read from ~/.aws/credentials, environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY), or passed explicitly via --access-key / --secret-key.


Usage — Azure Blob Storage

# Single blob (connection string)
pfc-migrate azure \
  --container my-logs \
  --blob archive/app_2026-03.jsonl.gz \
  --out-container my-logs-pfc \
  --out-prefix converted/ \
  --connection-string "DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;"

# All blobs matching a prefix
pfc-migrate azure \
  --container my-logs \
  --prefix archive/2026-03/ \
  --out-container my-logs-pfc \
  --out-prefix converted/2026-03/ \
  --format gz \
  --connection-string "..."

# Using account URL + DefaultAzureCredential (managed identity, az login, etc.)
pfc-migrate azure \
  --container my-logs \
  --prefix archive/ \
  --out-container my-logs-pfc \
  --account-url https://myaccount.blob.core.windows.net

--connection-string or --account-url required. With --account-url, DefaultAzureCredential is used automatically (managed identity, az login, environment variables).


Usage — Google Cloud Storage

# Single object
pfc-migrate gcs \
  --bucket my-logs \
  --blob archive/app_2026-03.jsonl.gz \
  --out-bucket my-logs-pfc \
  --out-prefix converted/

# All objects matching a prefix
pfc-migrate gcs \
  --bucket my-logs \
  --prefix archive/2026-03/ \
  --out-bucket my-logs-pfc \
  --out-prefix converted/2026-03/ \
  --format gz \
  --verbose

Credentials from GOOGLE_APPLICATION_CREDENTIALS, gcloud auth application-default login, or Workload Identity on GKE.


Lossless guarantee

Every conversion is verified by full decompression and MD5 check before the output is written. If anything doesn't match, the output file is deleted and the error is reported — the original is never modified unless --delete is explicitly passed.

The full test suite verifies 4 backends × 4 formats with:

  • Sorted MD5 of all 400,000 lines (identical to original ✅)
  • DuckDB random access: 39,743 lines in a time window, MD5 identical to original ✅

Project Description
pfc-jsonl Core binary — compress, decompress, query
pfc-duckdb DuckDB Community Extension (INSTALL pfc FROM community)
pfc-fluentbit Fluent Bit → PFC forwarder for live pipelines

License

MIT — see LICENSE.

Built by ImpossibleForge

Metadata

Release files for pfc-migrate 0.3.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for pfc-migrate 0.3.2
File Size Uploaded Provenance Actions
pfc_migrate-0.3.2.tar.gz 13.9 kB Without provenance Details for pfc_migrate-0.3.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for pfc-migrate 0.3.2
File Interpreter ABI Platform Actions
pfc_migrate-0.3.2-py3-none-any.whl Python 3 none any Details for pfc_migrate-0.3.2-py3-none-any.whl

Total release size: 27.4 kB

Release files / pfc_migrate-0.3.2.tar.gz

Download URL pfc_migrate-0.3.2.tar.gz
Size 13.9 kB
Tags Source
SHA-256 checksum
How to use checksums
cb9d7218c7df0c79753e2cf666afa80c4a04152ec9b3a2bc439149b3616d4af3
BLAKE2b-256 checksum
How to use checksums
92c8b226a5ee71099b08d472d581a1cd4992e58195212c1a8a2b2520e5957e16
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.2

Release files / pfc_migrate-0.3.2-py3-none-any.whl

Download URL pfc_migrate-0.3.2-py3-none-any.whl
Size 13.5 kB
Tags Python 3
SHA-256 checksum
How to use checksums
02a7a1c302a21913db29ac1ec25128a668195755769ff49267020ad3496672f1
BLAKE2b-256 checksum
How to use checksums
106e498527fff79d1aef3a2053bbd1719247f8eb467e774e6d6c751c1a459208
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.2.0 CPython/3.14.2

Security for pfc-migrate 0.3.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page