Skip to main content

GeoLibs Dator - A data extractor

Project description

GeoLibs-Dator

Dator, a data extractor (ETL as a library), that uses Pandas' DataFrames as in memory temporal storage.

Features

Source Extract Transform Load
BigQuery Y Y
CARTO Y Y Y*
CSV Y Y
Pandas Y
PostgreSQL Y Y Y

* Note: We are waiting for the append feature on CARTOframes, because the one we are using is a ñapa.

Configuration

Create a config.yml file using the config.example.yml one as guide. You can find in that one all the possible ETL cases.

If you are using BigQuery in your ETL process, you need to add a GOOGLE_APPLICATION_CREDENTIALS environment variable with the path to your Google Cloud's credentials.json file.

You can test them with the example.py file.

Example

from dator import Dator

dator = Dator('/usr/src/app/dator_config.yml')
df = dator.extract()
df = dator.transform(df)
dator.load(df)

TODOs

  • Better doc.
  • Tests.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

geolibs-dator-0.0.3.tar.gz (7.5 kB view hashes)

Uploaded Source

Built Distribution

geolibs_dator-0.0.3-py3-none-any.whl (10.7 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page