Skip to main content

Data migration utilities

Project description

Build Status Documentation Status

https://binstar.org/blaze/odo/badges/build.svg https://binstar.org/blaze/odo/badges/version.svg

Data migration in Python

Documentation

Example

Odo migrates data between different containers

>>> from odo import odo
>>> odo((1, 2, 3), list)
[1, 2, 3]

It operates on small, in-memory containers (as above) and large, out-of-core containers (as below)

>>> odo('myfile.hdf5::/data', 'postgresql://user:pass@host::my-table')
Table('my-table', MetaData(bind=Engine(postgresql://user:****@host)), ...)

Odo leverages the existing Python ecosystem. The example above uses sqlalchemy for SQL interation and h5py for HDF5 interaction.

Method

Odo migrates data using network of small data conversion functions between type pairs. That network is below:

odo conversions

Each node is a container type (like pandas.DataFrame or sqlalchemy.Table) and each directed edge is a function that transforms or appends one container into or onto another. We annotate these functions/edges with relative costs.

This network approach allows odo to select the shortest path between any two types (thank you networkx). For performance reasons these functions often leverage non-Pythonic systems like NumPy arrays or native CSV->SQL loading functions. Odo is not dependent on only Python iterators.

This network approach is also robust. When libraries go missing or runtime errors occur odo can work around these holes and find new paths.

This network approach is extensible. It is easy to write small functions and register them to the overall graph. In the following example showing how we convert from pandas.DataFrame to a numpy.ndarray.

from odo import convert

@convert.register(np.ndarray, pd.DataFrame, cost=1.0)
def dataframe_to_numpy(df, **kwargs):
    return df.to_records(index=False)

We decorate convert functions with the target and source types as well as a relative cost. This decoration establishes a contract that the underlying function must fulfill, in this case with the fast DataFrame.to_records method. Similar functions exist for append, to add to existing data, and resource for URI resolution.

  • convert: Transform dataset into new container

  • append: Add dataset onto existing container

  • resource: Given a URI find the appropriate data resource

  • odo: Call one of the above based on inputs. E.g. odo((1, 2, 3), list) -> convert(list, (1, 2, 3)) while L = []; odo((1, 2, 3), L) -> append(L, (1, 2, 3))

Finally, odo is also aware of which containers must reside in memory and which do not. In the graph above the red-colored nodes are robust to larger-than-memory datasets. Transformations between two out-of-core datasets operate only on the subgraph of the red nodes.

LICENSE

New BSD. See License File.

History

Odo was factored out from the Blaze project.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

odo-fixpandas20-0.5.3.tar.gz (141.4 kB view details)

Uploaded Source

Built Distribution

odo_fixpandas20-0.5.3-py2.py3-none-any.whl (138.5 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file odo-fixpandas20-0.5.3.tar.gz.

File metadata

  • Download URL: odo-fixpandas20-0.5.3.tar.gz
  • Upload date:
  • Size: 141.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.1.0 requests-toolbelt/0.8.0 tqdm/4.25.0 CPython/3.6.4

File hashes

Hashes for odo-fixpandas20-0.5.3.tar.gz
Algorithm Hash digest
SHA256 710f26d1bce33ce41ae9db5043f26525302f7df8dd89025e7a51024272bea3c5
MD5 bec86727ba601cd2d28fa987a4a9d0d2
BLAKE2b-256 f9a5e0c3716cd9ecfbafcc39735991ae46dd5e5fd507b0f9c0073792a7e447db

See more details on using hashes here.

File details

Details for the file odo_fixpandas20-0.5.3-py2.py3-none-any.whl.

File metadata

  • Download URL: odo_fixpandas20-0.5.3-py2.py3-none-any.whl
  • Upload date:
  • Size: 138.5 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.11.0 pkginfo/1.4.2 requests/2.19.1 setuptools/39.1.0 requests-toolbelt/0.8.0 tqdm/4.25.0 CPython/3.6.4

File hashes

Hashes for odo_fixpandas20-0.5.3-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 4488d93b51438b900d6fa44c83f6cabc0aed16955ec21a62a2d2f25541117a5a
MD5 a20903722f55a4cc6fac762b528228d2
BLAKE2b-256 0687d523c3f4b2a276df32f3cf2b5367fc1bebb65861710d2dfc1711536ab885

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page