Skip to main content

This library splits large dataframes into smaller chunks, which are then passed to multiprocessing

Project description

#MultiProcessDivision()

This package is an optimized approach to split large pd.DataFrame() or pd.Series() objects for optimized multiprocessing processing. The core aim is to provide a stable interface, which allows splitting vectorized objects along specified axes.
The class only consists of a single function, divide_df(), which requires the following parameters to work seamlessly:

  • The data provided for splitting. Data must be provided as pd.DataFrame() or pd.Series() objects.
  • The axis along which a split is to be conducted. While this parameter can either be set 0 (index) or 1 (columns) for pd.DataFrame() objects, it has to be set to 0 for pd.Series() objects.
  • In case a pd.Series() object is passed to the function, the "series" parameter has to be set to True.
  • The following "range_setter" parameter is optional. If it is not set, it defaults to None, which is the amount of logical cores, the executing system provides. Otherwise, if the amount of cores to execute the processing on is limited, the parameter must be set with a value smaller than the amount of logical cores.

##Questions and Feedback Please don't hesitate to provide me feedback, if you use the function in your stack. Improvements are warmly welcome.

Project details


Release history Release notifications | RSS feed

This version

1.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

MultiProcessDivision-1.0.tar.gz (3.1 kB view hashes)

Uploaded Source

Built Distribution

MultiProcessDivision-1.0-py3-none-any.whl (3.6 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page