Run pandas-profiling HTML reports for a given list of database tables.
Are you working with an unfamiliar database and feeling confused?
Do you find yourself running
SELECT * FROM TABLE LIMIT 100 queries to remind yourself possible values of key fields?
Do these values slowly change over the course of years, with no clear documentation from upstream data producers?
If you answered "yes" to any of the above questions, this package may be for you.
What it does
Poliwhirl helps you orient yourself in an unfamiliar database by generating useful HTML reports (via pandas-profiling) for key tables you specify. It saves all these outputs to a single directory, which you can index locally with Spotlight or Alfred, or even deploy to some internal static website for your team.
You can install this package via pip:
pip install polywhirl
Features and usage
Polywhirl takes a single argument of a yaml file containing the structure of the database you'd like to profile. The format of this yaml file approximately follows that of dbt's schema.yml. A template file
tables.yml is provided for you, but you'll need to input the lists of schemas and tables specific to your own database.
Polywhirl currently supports these connections:
- BigQuery (use
- Redshift (use
For the sake of performance, polywhirl will pull a random sample of 10k rows from each table. For Redshift, it supports defining a sortkey for each table, which is used to limit data to the most recent 90 days. This improves performance on large tables.
BigQuery credentials are handled by
pandas.read_gbq() which relies on the
Redshift credentials are requested on first run, then stored locally in your system keychain using the keyring package.
Do you realize you misspelled Poliwhirl?
Yes I noticed this as I was writing this README—but it's too late—the name has grown on me.
Would like to get to the following in the future, so feel free to send PRs my way on any of these:
- Add equivalent of Redshift sortkey to BigQuery logic, to prevent unnecessary full table scans
- Compile .html outputs into a searchable static website
- Automated tests (w/ pytest)
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size polywhirl-0.1.0-py3-none-any.whl (6.2 kB)||File type Wheel||Python version py3||Upload date||Hashes View hashes|
|Filename, size polywhirl-0.1.0.tar.gz (4.8 kB)||File type Source||Python version None||Upload date||Hashes View hashes|
Hashes for polywhirl-0.1.0-py3-none-any.whl