rdc.etl 1.0.0a6

Documentation

Full documentation for rdc.etl on ReadTheDocs.

This is a work in progress, the 1.0 API may change until 1.0 is released.

Example usage

>>> # Sample data extract transformation.
>>> # Use hardcoded data here for sample purpose.
>>> from rdc.etl.transform.extract import Extract
>>> @Extract
... def sample_extract():
...     yield {'first_name': 'John', 'last_name': 'Doe', }
...     yield {'first_name': 'Jane', 'last_name': 'Dae', }

>>> # Sample data transformation.
>>> from rdc.etl.transform import Transform
>>> @Transform
... def sample_transform(hash, channel):
...     hash['last_name'] = hash['last_name'].upper()
...     hash['initials'] = '{0}.{1}.'.format(hash['first_name'][0], hash['last_name'][0]).upper()
...     yield hash

>>> # Sample load. This is only a screen log for sample purpose.
>>> from rdc.etl.transform.util import Log
>>> sample_load_to_screen = Log()

>>> # Tie everything together, then run!
>>> from rdc.etl.harness.threaded import ThreadedHarness
>>> job = ThreadedHarness()
>>> job.add_chain(sample_extract, sample_transform, sample_load_to_screen)
>>> job()

Running the Test Suite

pip install nose
make test

Release Notes

Contributing

I’m Romain Dorgueil.

rdc.etl is on GitHub.

Get in touch, via GitHub or otherwise, if you’ve got something to contribute, it’d be most welcome!

If you feel overwhelmingly grateful, or want to support the project you can tip me on Gittip.