pypofatu

A python package to curate Pofatu data.

Installation

Install pypofatu from PyPI running

pip install pypofatu

preferably in a new virtual environment, to keep your system's Python installation unaffected.

Installing pypofatu will also install a command line program pofatu, which provides functionality to curate and query Pofatu data. Run

pofatu -h

for details on usage.

pypofatu operates on data as displayed by the Pofatu database. This data curated in a GitHub repository which is accessed by pypofatu locally. Thus, in order to use pypofatu you must clone the pofatu/pofatu-data repository or download a release version.

Curation workflow

Check the data in the repository for consistency:

pofatu --repos PATH/TO/REPOS check

Create a distribution of the data:

pofatu --repos PATH/TO/REPOS dist

A distribution consists of

• a set of CSV files described by CSVW metadata
• an SQLite database created from the CSVW package

Querying

Exploratory analysis of the data is possible with off-the-shelf tools such as grep, head etc. on the UNIX shell. More accurate analysis can be done with tools that exploit the CSV structure, e.g. the tools from the csvkit package. Ultimate control and performance is possible using the SQLite database.

The pofatu query command provides a convenience wrapper to query the database (once it has been created via pofatu dist):

Run

pofatu --repos tests/repos/ dist
pofatu --repos tests/repos/ query .schema

to see the SQL schema of the database or

pofatu --repos tests/repos/ query "select count(*) from 'measurements.csv'"
INFO    SQLite database at dist/pofatu.sqlite
 count(*)  
----------
 78

to run simple queries. More complex queries can be written as SQL to a text file and run with

cat query.sql | pofatu --repos tests/repos/ query --format pipe -