pandas hug
sometimes you need to embrace your data and get a little, or a lot, more of it into memory.
your column data types are rarely space efficient. most of the time this is because they were chosen by someone else, but sometimes its just a hassle to find the most space efficient types.
pandas-hug is here to help crush your data to fit in memory.
installation
pip install pandas-hug
usage
import pandas as pd
import pandas_hug
S = pd.Series([2**8])
A = pd.Series([f'a{i}' for i in range(100)])
M = pd.Series([42])
E = pd.Series(['a', 'b', 'c'] * 15)
df = pd.DataFrame({'S': S, 'A': A, 'M': M, 'E': E})
df.info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 100 entries, 0 to 99
Data columns (total 4 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 S 1 non-null float64
1 A 100 non-null object
2 M 1 non-null float64
3 E 45 non-null object
dtypes: float64(2), object(2)
memory usage: 3.2+ KB
df.convert_dtypes().hug().info()
<class 'pandas.core.frame.DataFrame'>
RangeIndex: 100 entries, 0 to 99
Data columns (total 4 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 S 1 non-null UInt16
1 A 100 non-null string
2 M 1 non-null UInt8
3 E 45 non-null category
dtypes: UInt16(1), UInt8(1), category(1), string(1)
memory usage: 1.6 KB
pandas-hug monkey-patches pandas.DataFrame and pandas.Series to add the hug() method.
you should call convert_dtypes() before hugging your data. this does useful things like converting float to int (pandas >=1.2.0, dec 2020) and object to string where appropriate.
Metadata
Release files for pandas-hug 0.14.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| pandas_hug-0.14.0-py3-none-any.whl | Python 3 | none | any | Details for pandas_hug-0.14.0-py3-none-any.whl |
Release files / pandas_hug-0.14.0-py3-none-any.whl
| Download URL | pandas_hug-0.14.0-py3-none-any.whl |
|---|---|
| Size | 5.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
8bbb26072e34362a45e45c09c2283829e389c1436bc211907d8ed4158ca30703
|
|
BLAKE2b-256 checksum How to use checksums |
750bd7343affcda64bca4385c89a06946f1e57ab4e3ae3734d9e6816c33200f3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.10.8
|
Security for pandas-hug 0.14.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.