pg-binny 0.0.3

Bins your dataframe columns into the Top ≤N categories, and "Other".

pg_binny

Discretize a whole dataframe into ≤N bins, using Top N categories.

%nbdev_hide

The discretize function handles discrete & continuous columns:

• Continuous columns are cut into N bins using supplied cutting function (defaults to qcut for quantile cuts.
• Categorical columns: take the Top N-1, with the rest tossed into "Other"

TODO: Describe and show the plot helpers too.

Install

conda install pg_binny

-or-

pip install pg_binny

-or (locally)-

pip install -e . (That's "pip install -e dot")

How to use

Make a sample dataframe.

import pandas as pd
import pg_binny as binny


dataset = 'car_crashes'
try:
    import seaborn as sns
    df = sns.load_dataset(dataset)
except ModuleNotFoundError:
    df = pd.read_csv(f'https://raw.githubusercontent.com/mwaskom/seaborn-data/master/{dataset}.csv')
df.sample(5)

<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	total	speeding	alcohol	not_distracted	no_previous	ins_premium	ins_losses	abbrev
19	15.1	5.738	4.530	13.137	12.684	661.88	96.57	ME
15	15.7	2.669	3.925	15.229	13.659	649.06	114.47	IA
35	14.1	3.948	4.794	13.959	11.562	697.73	133.52	OH
50	17.4	7.308	5.568	14.094	15.660	791.14	122.04	WY
43	19.4	7.760	7.372	17.654	16.878	1004.75	156.83	TX

Discretize with default bins

dfd = binny.discretize(df)
dfd.sample(5)

---------------------------------------------------------------------------

AttributeError                            Traceback (most recent call last)

<ipython-input-2-41b3e27056d4> in <module>
----> 1 dfd = binny.discretize(df)
      2 dfd.sample(5)
      3 


AttributeError: module 'pg_binny' has no attribute 'discretize'

dfd['speeding'].dtype

CategoricalDtype(categories=[(1.7910000000000001, 2.413], (2.413, 3.496], (3.496, 3.948], (3.948, 4.095], (4.095, 4.608], (4.608, 5.032], (5.032, 6.014], (6.014, 6.923], (6.923, 7.76], (7.76, 9.45]],
, ordered=True)

dfd['total'].dtype

CategoricalDtype(categories=[(5.899, 11.1], (11.1, 12.3], (12.3, 13.6], (13.6, 14.5], (14.5, 15.6], (15.6, 17.4], (17.4, 18.1], (18.1, 19.4], (19.4, 21.4], (21.4, 23.9]],
, ordered=True)

You can set the #bins and the cutting function (defaults to quantile cut, but you may prefer plain-old cut, or something else.

?binny.discretize

Signature:
binny.discretize(
    df,
    nbins=10,
    cut=<function qcut at 0x7fae29d843b0>,
    verbose=2,
    drop_useless=True,
)
Docstring:
Discretize columns in {df} to have at most {nbins} categories.
  * Categorical columns: take the Top n-1 plus "Other"
  * Continuous columns: cut into {nbins} using {cut}.

Returns a new discretized dataframe with the same column names.
Promotes discrete columns to categories.

Parameters
-----------
df: Dataframe to discretize
nbins: Max number of bins to use. May return fewer.
cut: Cutting method. Default `pd.qcut`. Consider pd.cut, or write your own.
verbose: 0: silent, 1: colnames, 2: (Default) top N for each column
drop_useless: Removes columns that have < 2 unique values.

Replaces numerical NA values with 'NA'.
File:      /Volumes/Peregrine/binny/pg_binny/core.py
Type:      function

Other functions

[x for x in dir(binny) if x[:2] not in ['__', 'pa', 'pd', 'rc']]

['autolabel',
 'clean_category',
 'discretize',
 'drop_singletons',
 'is_numeric',
 'isnum']

?binny.autolabel

Signature: binny.autolabel(ax, border=False) -> None
Docstring:
Label bars in a barplot {ax} with their height.
Thanks to matplotlib, composition.ai, and jsoma/chart.py.

TODO: how to label with their legend labels?
File:      /Volumes/Peregrine/binny/pg_binny/core.py
Type:      function

?binny.clean_category

Signature: binny.clean_category(df, col: str) -> None
Docstring:
Remove unused categories from df.col, inplace.
If not a category, do nothing.
File:      /Volumes/Peregrine/binny/pg_binny/core.py
Type:      function

?binny.is_numeric

Signature: binny.is_numeric(col: str)
Docstring:
Returns True iff already numeric, or can be coerced.
Usage: df.apply(is_numeric)
Usage: is_numeric(df['colname'])

Returns Boolean series.

From:
https://stackoverflow.com/questions/54426845/how-to-check-if-a-pandas-dataframe-contains-only-numeric-column-wise
File:      /Volumes/Peregrine/binny/pg_binny/core.py
Type:      function

History

pg_binny is an example extracting some frequently copy/pasted routines into a general purpose nbdev project.

Originally called binny because it bins things, that was already taken on PyPi (for... a project that bins things). The prefix pg is short for the project we were working on.

The routines and text are completely general.