Typed DataFrames

Pandas DataFrame subclasses that self-organize and serialize robustly.

Film = TypedDfs.typed("Film").require("name", "studio", "year").build()
df = Film.read_csv("file.csv")
assert df.columns.tolist() == ["name", "studio", "year"]
type(df)  # Film

Your types remember how to be read, including columns, dtypes, indices, and custom requirements. No index_cols=, header=, set_index, or astype needed.

Read and write any format:

path = input("input file? [.csv/.tsv/.tab/.json/.xml.bz2/.feather/.snappy.h5/...]")
df = Film.read_file(path)
df.write_file("output.snappy")

Need dataclasses?

instances = df.to_dataclass_instances()
Film.from_dataclass_instances(instances)

Save metadata?

df = df.set_attrs(dataset="piano")
df.write_file("df.csv", attrs=True)
df = Film.read_file("df.csv", attrs=True)
print(df.attrs)  # e.g. {"dataset": "piano")

Make dirs? Don’t overwrite?

df.write_file("df.csv", mkdirs=True, overwrite=False)

Write / verify checksums?

df.write_file("df.csv", file_hash=True)
df = Film.read_file("df.csv", file_hash=True)  # fails if wrong

Get example datasets?

print(ExampleDfs.penguins().df)
#    species     island  bill_length_mm  ...  flipper_length_mm  body_mass_g     sex
# 0    Adelie  Torgersen            39.1  ...              181.0       3750.0    MALE

Pretty-print the obvious way?

df.pretty_print(to="all_data.md.zip")
wiki_txt = df.pretty_print(fmt="mediawiki")

All standard DataFrame methods remain available. Use .of(df) to convert to your type, or .vanilla() for a plain DataFrame.

Read the docs 📚 for more info and examples.

🐛 Pandas serialization bugs fixed

Pandas has several issues with serialization.

See: Fixed issues

Depending on the format and columns, these issues occur:

columns being silently added or dropped,
errors on either read or write of empty DataFrames,
the inability to use DataFrames with indices in Feather,
writing to Parquet failing with half-precision,
lingering partially written files on error,
the buggy xlrd being preferred by read_excel,
the buggy odfpy also being preferred,
writing a file and reading it back results in a different DataFrame,
you can’t write fixed-width format,
and the platform text encoding being used rather than utf-8.
invalid JSON is written via the built-in json library

🎁 Other features

See more in the guided walkthrough ✏️

See: Short feature list

Dtype-aware natural sorting
UTF-8 by default
Near-atomicity of read/write
Matrix-like typed dataframes and methods (e.g. matrix.is_symmetric())
DataFrame-compatible frozen, hashable, ordered collections (dict, list, and set)
Serialize JSON robustly, preserving NaN, inf, −inf, enums, timezones, complex numbers, etc.
Serialize more formats like TOML and INI
Interpreting paths and formats (e.g. FileFormat.split("dir/myfile.csv.gz").compression # gz)
Generate good CLI help text for input DataFrames
Parse/verify/add/update/delete files in a .shasum-like file

💔 Limitations

See: List of limitations

Multi-level columns are not yet supported.
Columns and index levels cannot share names.
Duplicate column names are not supported. (These are strange anyway.)
A typed DF cannot have columns "level_0", "index", or "Unnamed: 0".
inplace is forbidden in some functions; avoid it or use .vanilla().

🔌 Serialization support

TypedDfs provides the methods read_file and write_file, which guess the format from the filename extension. For example, this will convert a gzipped, tab-delimited file to Feather:

TastyDf = typeddfs.typed("TastyDf").build()
TastyDf.read_file("myfile.tab.gz").write_file("myfile.feather")

Pandas does most of the serialization, but some formats require extra packages. Typed-dfs specifies extras to help you get required packages and with compatible versions.

Here are the extras:

feather: Feather (uses: pyarrow)
parquet: Parquet (e.g. .snappy) (uses: pyarrow)
xml (uses: lxml)
excel: Excel and LibreOffice .xlsx/.ods/.xls, etc. (uses: openpyxl, defusedxml)
toml: TOML (uses: tomlkit)
html (uses: html5lib, beautifulsoup4)
xlsb: rare binary Excel file (uses: pyxlsb)
HDF5 {no extra provided} (use: tables)

For example, for Feather and TOML support use: typeddfs[feather,toml]
As a shorthand for all formats, use typeddfs[all].

📊 Serialization in-depth

See: Full table

format	packages	extra	sanity	speed	file sizes
Feather	`pyarrow`	`feather`	+++	++++	+++
Parquet	`pyarrow` or `fastparquet` †	`parquet`	++	+++	++++
csv/tsv	none	none	++	−−	−−
flexwf ‡	none	none	++	−−	−−
.fwf	none	none	+	−−	−−
json	none	none	−−	−−−	−−−
xml	`lxml`	`xml`	−	−−−	−−−
.properties	none	none	−−	−−	−−
toml	`tomlkit`	`toml`	−−	−−	−−
INI	none	none	−−−	−−	−−
.lines	none	none	++	−−	−−
.npy	none	none	−	+	+++
.npz	none	none	−	+	+++
.html	`html5lib,beautifulsoup4`	`html`	−−	−−−	−−−
pickle	none	none	−−	−−−	−−−
XLSX	`openpyxl,defusedxml`	`excel`	+	−−	+
ODS	`openpyxl,defusedxml`	`excel`	+	−−	+
XLS	`openpyxl,defusedxml`	`excel`	−−	−−	+
XLSB	`pyxlsb`	`xlsb`	−−	−−	++
HDF5	`tables`	`hdf5`	−−	−	++

⚠ Note: The hdf5 extra is currently disabled.

See: serialization notes

† fastparquet can be used instead. It is slower but much smaller.
Parquet only supports str, float64, float32, int64, int32, and bool. Other numeric types are automatically converted during write.
‡ .flexwf is fixed-width with optional delimiters.
JSON has inconsistent handling of None. (orjson is more consistent).
XML requires Pandas 1.3+.
Not all JSON, XML, TOML, and HDF5 files can be read.
.ini and .properties can only be written with exactly 2 columns + index levels: a key and a value. INI keys are in the form section.name.
.lines can only be written with exactly 1 column or index level.
.npy and .npz only serialize numpy objects. They are not supported in read_file and write_file.
.html is not supported in read_file and write_file.
Pickle is insecure and not recommended.
Pandas supports odfpy for ODS and xlrd for XLS. In fact, it prefers those. However, they are very buggy; openpyxl is much better.
XLSM, XLTX, XLTM, XLS, and XLSB files can contain macros, which Microsoft Excel will ingest.
XLS is a deprecated format.
XLSB is not fully supported in Pandas.
HDF may not work on all platforms yet due to a tables issue.

Feather offers massively better performance over CSV, gzipped CSV, and HDF5 in read speed, write speed, memory overhead, and compression ratios. Parquet typically results in smaller file sizes than Feather at some cost in speed. Feather is the preferred format for most cases.

🔒 Security

Refer to the security policy.

📝 Extra notes

See: Pinned versions

Dependencies in the extras only have version minimums, not maximums. For example, typed-dfs requires pyarrow >= 4. natsort is also only assigned a minimum version number. This means that the result of typed-df’s sort_natural could change. To fix this, pin natsort to a specific major version; e.g. natsort = "^8" with Poetry or natsort>=8,<9 with pip.

🍁 Contributing

Typed-Dfs is licensed under the Apache License, version 2.0. New issues and pull requests are welcome. Please refer to the contributing guide. Generated with Tyrannosaurus.

Metadata

Release files for typeddfs 0.16.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for typeddfs 0.16.5
File	Size	Uploaded	Provenance
typeddfs-0.16.5.tar.gz	77.9 kB	Mar 1, 2022	Without provenance	Details

Built distribution (wheel)

Table of built distributions (wheels) for typeddfs 0.16.5
File	Interpreter	ABI	Platform	Reset
typeddfs-0.16.5-py3-none-any.whl 92.4 kB Mar 1, 2022 Without provenance	Python 3	none	any	Details

Total release size: 170.3 kB

Release files / typeddfs-0.16.5.tar.gz

Download URL	typeddfs-0.16.5.tar.gz
Size	77.9 kB
Tags	Source
SHA-256 checksum How to use checksums	`631296a252ddb614c997596d75b268768ab6c225becc434311f79e9027d73d15`
BLAKE2b-256 checksum How to use checksums	`df6923b4c90de17493d82dcd65f094b636dc389fc3cd99342a109678bb06e101`
Upload date	Mar 1, 2022
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`poetry/1.1.13 CPython/3.9.10 Linux/5.11.0-1028-azure`

Release files / typeddfs-0.16.5-py3-none-any.whl

Download URL	typeddfs-0.16.5-py3-none-any.whl
Size	92.4 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`6c2a0a98a2bdd6bb941219864abc22874e30d1aa720e4e86dc77061db863ee98`
BLAKE2b-256 checksum How to use checksums	`00d206bf2e9884a5758e0a999504a48462ff40a8777f3a602f83611e011fdb55`
Upload date	Mar 1, 2022
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`poetry/1.1.13 CPython/3.9.10 Linux/5.11.0-1028-azure`

Security for typeddfs 0.16.5

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.