dfsummarizer

This is an application to summarize the variables in a data frame. It will accept a CSV, TSV or XLS file and produce a table summarizing all columns individually.

This was motivated by the fact that the summary function for a pandas data frame ignores all non-numeric columns, and does not contain multiple common analytical considerations: how many unique values, how many missing values, min and max dates, min, mean and max string lengths.

Output can be generated as either Latex or Markdown.

Released and distributed via setuptools/PyPI/pip for Python 3.

Additional detail available in the companion blog post

Notes

Initial implementation can handle larger files by chunking data and iteratively building statistics. All statistics are robust except for estimation of the proportion of unique values. We have used a simple implementation of the Flajolet Martin algorithm based on the implementation by Javia Jinkal

This review article by Phillip Gibbons gives a great overview of the alternatives.

Usage

You can use this application multiple ways

Use the runner:

./dfsummarizer-runner.py markdown data/test.csv > markdown_test.md

Which was used to generate the markdown output test file

Invoke the directory as a package:

python -m dfsummarizer markdown data/test.csv

Or simply install the package and use the command line application directly

Installation

Installation from the source tree:

python setup.py install

(or via pip from PyPI):

pip install dfsummarizer

Now, the dfsummarizer command is available::

dfsummarizer markdown test.csv

This will produce a markdown table summarizing the contents of the CSV file test.csv

Acknowledgements

Python package built using the bootstrap cmdline template by jgehrcke