poppusher

What is popgetter?

Popgetter is a collection of tools, designed to make it convenient to download census data from a number of different jurisdictions and coercing the data into common formats. The aim is that city or region scale analysis can be easily replicated for different geographies, using the most detailed, locally available data.

What is poppusher?

This repo is "poppusher", which is one component of the popgetter project. Poppusher is a pipeline which downloads data from a number of different jurisdictions and then processes it into a common format. The data is then stored in a cloud-based data store, which can be accessed by other components of the popgetter project.

See the flow diagram for more details.

What the popgetter system does and doesn't do

Popgetter DOES:

For each of the implemented countries:

• Download the most detailed geometries, for which census data is available.
• Download the most detailed census available for most, if not all, variables published by the census.
• Ensures that the geometries and census data join correctly.
• Presents some standard metadata to allow the user to explore which variables are available.
• publish the data in a set of common file types (eg CloudGeoBuff, Parquet, PMtiles).

Popgetter DOES NOT:

• It does not attempt to ensure that census variables are comparable between different jurisdictions. Nor does it attempt to ensure that the results of any analysis can be directly compared across multiple countries.

Getting started

At present, this is still a development project, so the first step is to clone the repo and then install using pip, with the --editable option:

1. Create a virtual environment and activate it (you should be able to use your own choice of environment manager, such as conda or venv, but so far pyenv is the most tested with popgetter). eg:

python  -m venv poppusher_venv    # create a virtual environment called `poppusher_venv`
source poppusher_venv/bin/activate  # activate the virtual environment

2. Clone the repo and then do an 'editable' install:

git clone https://github.com/Urban-Analytics-Technology-Platform/poppusher.git
cd poppusher
pip install -e ".[dev]"

3. Then, start the Dagster UI web server:

dagster dev

Open http://localhost:3000 with your browser to see the project.

Development

You can start writing assets in poppusher/assets/ directory. New assets and jobs will need to be added to the poppusher/__init__.py file.

Adding new Python dependencies

You can specify new Python dependencies in pyproject.toml.

Unit testing

Tests are in the poppusher_tests directory and you can run tests using pytest:

pytest

Repo structure

This is a Dagster project. The repo layout was initially created with the dagster project scaffold command. It has been subsequently updated using the copier command and the Scientific Python template.

There is code, which predates the migration to Dagster in the previous_code directory. In due course, this will be removed as the remaining countries are migrated to Dagster. There are usage instructions for this old code in previous_code/previous_code_usage.md.