pg-binny 0.0.1

Bins your dataframe columns into the Top ≤N categories, and "Other".

pg_binny

Discretize a whole dataframe into ≤N bins, using Top N categories.

The discretize function handles discrete & continuous columns:

• Continuous columns are cut into N bins using supplied cutting function (defaults to qcut for quantile cuts.
• Categorical columns: take the Top N-1, with the rest tossed into "Other"

TODO: Describe and show the plot helpers too.

Install

conda install pg_binny

-or-

pip install pg_binny

-or (locally)-

pip install -e . (That's "pip install -e dot")

How to use

Make a sample dataframe.

import numpy as np
import pandas as pd

dataset = 'car_crashes'
try:
    import seaborn as sns
    df = sns.load_dataset(dataset)
except ModuleNotFoundError:
    df = pd.read_csv(f'https://raw.githubusercontent.com/mwaskom/seaborn-data/master/{dataset}.csv')
df.sample(5)

<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	total	speeding	alcohol	not_distracted	no_previous	ins_premium	ins_losses	abbrev
12	15.3	5.508	4.437	13.005	14.994	641.96	82.75	ID
45	13.6	4.080	4.080	13.056	12.920	716.20	109.61	VT
2	18.6	6.510	5.208	15.624	17.856	899.47	110.35	AZ
47	10.6	4.452	3.498	8.692	9.116	890.03	111.62	WA
1	18.1	7.421	4.525	16.290	17.014	1053.48	133.93	AK

Discretize with default bins

dfd = binny.discretize(df)
dfd.sample(5)

total:
	(5.899, 11.1]    6
	(11.1, 12.3]     5
	(12.3, 13.6]     6
	(13.6, 14.5]     4
	(14.5, 15.6]     5
	(15.6, 17.4]     5
	(17.4, 18.1]     5
	(18.1, 19.4]     6
	(19.4, 21.4]     5
	(21.4, 23.9]     4
speeding:
	(1.7910000000000001, 2.413]    6
	(2.413, 3.496]                 5
	(3.496, 3.948]                 5
	(3.948, 4.095]                 5
	(4.095, 4.608]                 5
	(4.608, 5.032]                 5
	(5.032, 6.014]                 5
	(6.014, 6.923]                 5
	(6.923, 7.76]                  5
	(7.76, 9.45]                   5
alcohol:
	(1.592, 3.328]     6
	(3.328, 3.567]     5
	(3.567, 3.948]     5
	(3.948, 4.272]     5
	(4.272, 4.554]     5
	(4.554, 4.968]     5
	(4.968, 5.456]     5
	(5.456, 5.655]     5
	(5.655, 6.765]     5
	(6.765, 10.038]    5
not_distracted:
	(1.7590000000000001, 8.576]    6
	(8.576, 9.944]                 5
	(9.944, 10.92]                 5
	(10.92, 13.056]                5
	(13.056, 13.857]               5
	(13.857, 14.35]                5
	(14.35, 15.624]                5
	(15.624, 16.692]               5
	(16.692, 18.308]               5
	(18.308, 23.661]               5
no_previous:
	(5.899, 8.856]      6
	(8.856, 10.848]     5
	(10.848, 11.592]    5
	(11.592, 12.92]     5
	(12.92, 13.775]     5
	(13.775, 15.13]     5
	(15.13, 16.038]     5
	(16.038, 17.014]    5
	(17.014, 18.706]    5
	(18.706, 21.28]     5
ins_premium:
	(641.9590000000001, 688.75]    6
	(688.75, 732.28]               5
	(732.28, 780.45]               5
	(780.45, 804.71]               5
	(804.71, 858.97]               5
	(858.97, 881.51]               5
	(881.51, 913.15]               5
	(913.15, 1048.78]              5
	(1048.78, 1148.99]             5
	(1148.99, 1301.52]             5
ins_losses:
	(82.749, 106.62]    6
	(106.62, 110.35]    5
	(110.35, 120.21]    5
	(120.21, 133.35]    5
	(133.35, 136.05]    5
	(136.05, 142.39]    5
	(142.39, 148.58]    5
	(148.58, 153.72]    5
	(153.72, 159.85]    5
	(159.85, 194.78]    5
abbrev:
	AL        1
	AR        1
	GA        1
	ID        1
	MA        1
	NJ        1
	NV        1
	OR        1
	UT        1
	VT        1
	Other    41
  DROPPED [] because < 2 vals each.

<style scoped> .dataframe tbody tr th:only-of-type { vertical-align: middle; }

.dataframe tbody tr th {
    vertical-align: top;
}

.dataframe thead th {
    text-align: right;
}

</style>

	total	speeding	alcohol	not_distracted	no_previous	ins_premium	ins_losses	abbrev
35	(13.6, 14.5]	(3.496, 3.948]	(4.554, 4.968]	(13.857, 14.35]	(10.848, 11.592]	(688.75, 732.28]	(133.35, 136.05]	Other
28	(14.5, 15.6]	(5.032, 6.014]	(4.554, 4.968]	(13.857, 14.35]	(13.775, 15.13]	(913.15, 1048.78]	(136.05, 142.39]	NV
41	(18.1, 19.4]	(5.032, 6.014]	(5.655, 6.765]	(18.308, 23.661]	(16.038, 17.014]	(641.9590000000001, 688.75]	(82.749, 106.62]	Other
6	(5.899, 11.1]	(4.608, 5.032]	(3.567, 3.948]	(8.576, 9.944]	(5.899, 8.856]	(1048.78, 1148.99]	(159.85, 194.78]	Other
15	(15.6, 17.4]	(2.413, 3.496]	(3.567, 3.948]	(14.35, 15.624]	(12.92, 13.775]	(641.9590000000001, 688.75]	(110.35, 120.21]	Other

dfd['speeding'].dtype

CategoricalDtype(categories=[(1.7910000000000001, 2.413], (2.413, 3.496], (3.496, 3.948], (3.948, 4.095], (4.095, 4.608], (4.608, 5.032], (5.032, 6.014], (6.014, 6.923], (6.923, 7.76], (7.76, 9.45]],
, ordered=True)

dfd['total'].dtype

CategoricalDtype(categories=[(5.899, 11.1], (11.1, 12.3], (12.3, 13.6], (13.6, 14.5], (14.5, 15.6], (15.6, 17.4], (17.4, 18.1], (18.1, 19.4], (19.4, 21.4], (21.4, 23.9]],
, ordered=True)

You can set the #bins and the cutting function (defaults to quantile cut, but you may prefer plain-old cut, or something else.

?binny.discretize

Signature:
binny.discretize(
    df,
    nbins=10,
    cut=<function qcut at 0x7fae29d843b0>,
    verbose=2,
    drop_useless=True,
)
Docstring:
Discretize columns in {df} to have at most {nbins} categories.
  * Categorical columns: take the Top n-1 plus "Other"
  * Continuous columns: cut into {nbins} using {cut}.

Returns a new discretized dataframe with the same column names.
Promotes discrete columns to categories.

Parameters
-----------
df: Dataframe to discretize
nbins: Max number of bins to use. May return fewer.
cut: Cutting method. Default `pd.qcut`. Consider pd.cut, or write your own.
verbose: 0: silent, 1: colnames, 2: (Default) top N for each column
drop_useless: Removes columns that have < 2 unique values.

Replaces numerical NA values with 'NA'.
File:      /Volumes/Peregrine/binny/pg_binny/core.py
Type:      function

Other functions

[x for x in dir(binny) if x[:2] not in ['__', 'pa', 'pd', 'rc']]

['autolabel',
 'clean_category',
 'discretize',
 'drop_singletons',
 'is_numeric',
 'isnum']

?binny.autolabel

Signature: binny.autolabel(ax, border=False) -> None
Docstring:
Label bars in a barplot {ax} with their height.
Thanks to matplotlib, composition.ai, and jsoma/chart.py.

TODO: how to label with their legend labels?
File:      /Volumes/Peregrine/binny/pg_binny/core.py
Type:      function

?binny.clean_category

Signature: binny.clean_category(df, col: str) -> None
Docstring:
Remove unused categories from df.col, inplace.
If not a category, do nothing.
File:      /Volumes/Peregrine/binny/pg_binny/core.py
Type:      function

?binny.is_numeric

Signature: binny.is_numeric(col: str)
Docstring:
Returns True iff already numeric, or can be coerced.
Usage: df.apply(is_numeric)
Usage: is_numeric(df['colname'])

Returns Boolean series.

From:
https://stackoverflow.com/questions/54426845/how-to-check-if-a-pandas-dataframe-contains-only-numeric-column-wise
File:      /Volumes/Peregrine/binny/pg_binny/core.py
Type:      function

History

pg_binny is an example extracting some frequently copy/pasted routines into a general purpose nbdev project.

Originally called binny because it bins things, that was already taken on PyPi (for... a project that bins things). The prefix pg is short for the project we were working on.

The routines and text are completely general.