Skip to main content

Parallel parser for large files

Project description

About

ParPar (Parallel Parser) is a light tool which makes it easy to distribute a function across a large file.

ParPar is meant to work on serialized data where some values are highly repeated across records for a given field. e.g.

a 1 1
a 1 2
a 2 3
a 2 4
a 3 5
a 3 6
b 1 7
b 1 8
b 2 9
b 2 10
b 3 11
b 3 12

although we have 12 records, for column 1 there are only 2 unique values. Likewise, for column 2 there are only 3 unique values. We could break this file up into smaller files under a directory:

<out-dir>/<col-1-value>/<col-2-value>

or vis versa.

How to use.

  1. Start by importing the ParPar class:
from parpar import ParPar
  1. Initialize an instance:
ppf = ParPar()
  1. Shard a large file into sub-files:
# shard by columns
ppf.shard(
  <input-file>, <output-directory>,
  <columns>, <delim>, <newline>
)

# shard by lines
ppf.shard_by_lines(
  <input-file>, <output-directory>,
  <number_of_lines>, <delim>, <newline>
)
  1. Check to make sure the number of records are the same:
files = ppf.shard_files(<output-directory>)
records = ppf.sharded_records(files)

from parpar.utils import filelines

print(records == filelines(<input-file>))
  1. Map an arbitrary function across each line of all shared files:
def foo(line, *args, **kwargs):
    pass

args = [1, 2, 3]
kwargs = {'a': 'x', 'b': 'y'}

ppf.shard_line_apply(<output-directory>,
  foo, args, kwargs,
  processes=<number-of-processes>
)
  1. Map an arbitrary function across all shared files:
def bar(file, *args, **kwargs):
    pass

args = [1, 2, 3]
kwargs = {'a': 'x', 'b': 'y'}

ppf.shard_file_apply(<output-directory>,
  bar, args, kwargs,
  processes=<number-of-processes>
)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

parpar-0.0.19.tar.gz (7.0 kB view hashes)

Uploaded source

Built Distribution

parpar-0.0.19-py3-none-any.whl (8.0 kB view hashes)

Uploaded py3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page