Skip to main content

Build a better understanding of your data in PostgreSQL.

Project description

Data Fluent for PostgreSQL

Documentation Status license PyPI - Python Version

Build a better understanding of your data in PostgreSQL.

The following shows an example report generated by this tool. It gives the numbers of rows, columns, bytes as well as human-friendly size counts for each table within a given PostgreSQL database.

The Metrics Report

The following shows the row count for every column that represents a date grouped by year and month.

The Time Distribution Report


On Ubuntu 20:

$ wget -qO- \ \
        | sudo apt-key add -
$ echo "deb xenial-pgdg main" \
    | sudo tee /etc/apt/sources.list.d/pgdg.list

$ sudo apt update
$ sudo apt install \
    git \
    python3-pip \
    python3-virtualenv \
    postgresql-13 \

On macOS:

$ brew install \
    git \
    postgresql \

Then, regardless of platform, setup a virtual environment and install this software package.

$ virtualenv ~/.fluency
$ source ~/.fluency/bin/activate
$ python3 -m pip install datafluent

Example Analysis

Clone FiveThirtyEight's data repository. It has a large number of CSV-formatted datasets they've used for some of their articles and graphics.

$ git clone ~/538data

Make sure you can access a PostgreSQL database on your machine. Below I'll grant access to my account on my Ubuntu 20 machine. Please adjust the username and password for your system.

$ sudo -u postgres \
    bash -c "psql -c \"CREATE USER mark
                       WITH PASSWORD 'test'

With access setup, I've created a PostgreSQL database called intel.

$ createdb intel

There are a few dependencies for csvkit that are required to be installed ahead of time.

To install Ubuntu 20's dependencies for csvkit run:

$ sudo apt-get install \
    libicu-dev \

To install macOS's dependencies for csvkit run:

$ brew install icu4c

Then, regardless of platform, run:

$ python3 -m pip install csvkit

I'll import one of the datasets within FiveThirtyEight's repository. Note, because the dates within this dataset are not formatted in YYYY-MM-DD format, I needed to override the date parsing so that the MM/DD/YYYY format would be inferred by csvsql properly.

$ csvsql --db postgresql:///intel \
         --insert ~/538data/congress-generic-ballot/generic_topline_historical.csv \

I'll then run Data Fluent which will generate a report in Excel format.

$ datafluent --url postgresql:///intel

The above will produce a fluency.xlsx file with two worksheets: Metrics and Time Distributions.

Further Help

The database URL is passed through to sqlalchemy. Please see their documentation on Database URLs for more information on their syntax and the drivers supported.

If you need to override any parameters in this application, please refer to the help documentation generated by typer:

$ datafluent --help
Usage: datafluent [OPTIONS]

  --url TEXT                      [default: postgresql://localhost:5432/intel]
  --output TEXT                   [default: fluency.xlsx]
  --install-completion [bash|zsh|fish|powershell|pwsh]
                                  Install completion for the specified shell.
  --show-completion [bash|zsh|fish|powershell|pwsh]
                                  Show completion for the specified shell, to
                                  copy it or customize the installation.

  --help                          Show this message and exit.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

datafluent-0.0.32.tar.gz (6.1 kB view hashes)

Uploaded Source

Built Distribution

datafluent-0.0.32-py3-none-any.whl (6.0 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page