Skip to main content

Python ETL library

Project description

Bamboo

Bamboo Logo

Build Status PyPI version Docs

Bamboo is a library intended to facilitate the process of creating extract, transform, and load (ETL) data pipelines. Bamboo includes many features such as the ability to download and cache HTTP assets, copy files from remote servers, run commands on remote servers, handle zipped archives, perform bulk database ingests and more.

Installation

pip install bamboo-lib

Additional Steps

If you will need to use the distributed locking functionality, you will need to install some additional software. Below are the instructions for macOS

Installing Sherlock on macOS.

brew install libmemcached

pip install pylibmc --install-option="--with-libmemcached=/usr/local/Cellar/libmemcached/1.0.18_2"

pip install sherlock

Running tests

To run the tests, simply run: pytest

Alternatively, if you would like to display all log/print statements run: pytest -s

Configuration

To change the default folder where Bamboo will store HTTP downloads, set the BAMBOO_DOWNLOAD_FOLDER environment variable. By default, downloads will go to /tmp.

To change the default logger settings, set BAMBOO_LOGGER_CONF to point to a logging configuration file (see example in example/logging.conf).

Troubleshooting tips

Mutli-processing issues on macOS

If you are running High Sierra or later and get an error like:

objc[30911]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug

Try setting:

export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for bamboo-lib, version 0.0.31
Filename, size File type Python version Upload date Hashes
Filename, size bamboo_lib-0.0.31-py3-none-any.whl (55.2 kB) File type Wheel Python version py3 Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page