Skip to main content

This project provides a collection of utilities for doing lightweight data wrangling.

Project description

datashaper

This project provides a collection of utilities for doing lightweight data wrangling.

There are two goals of the project:

  1. Create a shareable client/server schema for serialized wrangling instructions
  2. Maintain an implementation of a basic wrangling engine (based on Arquero) and in the case of python implemented in Pandas

Building

  • You need to install poetry python package manager.
  • Run: poetry install

Usage

This project is intended to be used as a library for lightweight data wrangling. In the examples folder there is a Notebook which provides several examples of how to create data wrangling pipelines and how to read json specifications that can be generated by the js implementation.

Example of joining two tables:

from datashaper.pipeline import Pipeline
import datashaper.types as types
import pandas as pd

# id   name
# 1    bob
# 2    joe
# 3    jane
parents = pd.DataFrame({
    "id": [1, 2, 3],
    "name": ['bob', 'joe', 'jane']
})

# id   kid
# 1    billy
# 1    jill
# 2    kaden
# 2    kyle
# 3    moe
kids = pd.DataFrame({
    "id": [1, 1, 2, 2, 3],
    "kid": ['billy', 'jill', 'kaden', 'kyle', 'moe']
})

pipeline = Pipeline()

pipeline.add_dataset('parents', parents)
pipeline.add_dataset('kids', kids)

pipeline.add(Step(
    verb=Verb.join,
    input="parents",
    output="output",
    args={
        "other": "kids",
        "on":["id"]
    }
))

# id   name    kid
# 1    bob     billy
# 1    bob     jill
# 2    joe     kaden
# 2    joe     kyle
# 3    jane    moe
result = pipeline.run()

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

datashaper-0.0.43.tar.gz (36.0 kB view details)

Uploaded Source

Built Distribution

datashaper-0.0.43-py3-none-any.whl (71.6 kB view details)

Uploaded Python 3

File details

Details for the file datashaper-0.0.43.tar.gz.

File metadata

  • Download URL: datashaper-0.0.43.tar.gz
  • Upload date:
  • Size: 36.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for datashaper-0.0.43.tar.gz
Algorithm Hash digest
SHA256 60c9f0877555ddf1cad8e67e5ff9c1f33c785d99f6a004590cc5ee70847a2152
MD5 890f6b58ecdbfb4a365443cce23e0ba9
BLAKE2b-256 fbb3bcdbf24709e7a22f8ce9f2ba8daaa854de06bef573b7aec28c7492efe951

See more details on using hashes here.

File details

Details for the file datashaper-0.0.43-py3-none-any.whl.

File metadata

  • Download URL: datashaper-0.0.43-py3-none-any.whl
  • Upload date:
  • Size: 71.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/5.0.0 CPython/3.12.2

File hashes

Hashes for datashaper-0.0.43-py3-none-any.whl
Algorithm Hash digest
SHA256 4e7b4ace1afd14ce5bd7282c2ea2a11a9cd8788b3bd58b4aef7f47662df2a61a
MD5 28f202cd50c2309ba0ff56288aba9f63
BLAKE2b-256 5d04ab99839b95d15bdb4b70b97338216d1fcf75616672e219d0d08546ce0fcf

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page