Skip to main content

Parallel parser for large files

Project description


ParPar (Parallel Parser) is a light tool which makes it easy to distribute a function across a large file.

ParPar is meant to work on serialized data where some values are highly repeated across records for a given field. e.g.

a 1 1
a 1 2
a 2 3
a 2 4
a 3 5
a 3 6
b 1 7
b 1 8
b 2 9
b 2 10
b 3 11
b 3 12

although we have 12 records, for column 1 there are only 2 unique values. Likewise, for column 2 there are only 3 unique values. We could break this file up into smaller files under a directory:


or vis versa.

How to use.

  1. Start by importing the ParPar class:
from parpar import ParPar
  1. Initialize an instance:
ppf = ParPar()
  1. Shard a large file into sub-files:
# shard by columns
  <input-file>, <output-directory>,
  <columns>, <delim>, <newline>

# shard by lines
  <input-file>, <output-directory>,
  <number_of_lines>, <delim>, <newline>
  1. Check to make sure the number of records are the same:
files = ppf.shard_files(<output-directory>)
records = ppf.sharded_records(files)

from parpar.utils import filelines

print(records == filelines(<input-file>))
  1. Map an arbitrary function across each line of all shared files:
def foo(line, *args, **kwargs):

args = [1, 2, 3]
kwargs = {'a': 'x', 'b': 'y'}

  foo, args, kwargs,
  1. Map an arbitrary function across all shared files:
def bar(file, *args, **kwargs):

args = [1, 2, 3]
kwargs = {'a': 'x', 'b': 'y'}

  bar, args, kwargs,

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

parpar-0.0.19.tar.gz (7.0 kB view hashes)

Uploaded source

Built Distribution

parpar-0.0.19-py3-none-any.whl (8.0 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page