Skip to main content

Cleaning your messy data.

Getting started

Consider cleaning up some messy data. Here is a deep nested dictionary containing lots of unnecessary nesting and tuple.

some_messy_data = {
    "body": {
        "article": {
             "articlesbody": {
                 "articlesmeta": {
                     "articles_meta_3": "Monty Python",
                 }
             }
        },
    },
    "published": {
        "datetime": ("2014-11-05", "23:00:00"),
    }
}

Values you want are 'Monty Python' and '2014-11-05', should be named 'title' and 'published_date'

Now let the hack begin with the dripper.

  • Defile declaration dictionary

  • Create dripper object by dripper.dripper_factory

  • Drip essential data

# Define
 declaration = {
    "title": ("body", "article", "articlesbody", "articlesmeta", "articles_meta_3"),
    "published_date": ("published", "datetime", 0)
}

# Create
import dripper
d = dripper.dripper_factory(declaration)

# And drip
dripped = d(some_messy_data)

assert dripped == {
    "title": "Monty Python",
    "published_date": "2014-11-05",
}

Installation

Just use pip to install

pip install dripper

Requirements

dripper won’t require any kind of outer libraries. Supporting Python versions are:

  • Python 2.7

  • Python 3.3

  • Python 3.4

  • Python 3.5

Basics

Above example is not all features of dripper. It is created to handle various data to clean up.

As value

from dripper import dripper_factory
declaration = {
    "title": ("meta", "meta1")
})
d = dripper_factory(declaration)
d({"meta": {"meta1": "Monty Python"}}) == {"title": "Monty Python"}

Also you can specify string or integer directly. It is as same as one-element tuple.

from dripper import dripper_factory
declaration = {
    "title": "meta"
})
d = dripper_factory(declaration)
d({"meta": "Monty Python"}) == {"title": "Monty Python"}

As dict

dripper can define nested dictionary. Just pass nested dictionary to dripper_factory.

from dripper import dripper_factory
declaration = {
    "article": {
        "title": ["meta", "meta1"],
    }
})
d = dripper_factory(declaration)
d({
    "meta": {
        "meta1": "Monty Python",
    },
}) == {
    "article": {
        "title": "Monty Python",
    }
}

You can apply '__source_root__' to set root path for dripping.

declaration = {
    "article": {
        "__source_root__": ("body", "meta"),
...
        "title": "meta1",
        "author": ("meta2", "meta22"),
    }
})
d = dripper_factory(declaration)
d({
    "body": {
        "meta": {
            "meta1": "Monty Python",
            "meta2": {"meta22": "John Due"}
        }
    }
}) == {
    "article": {
        "title": "Monty Python",
        "author": "John Due",
    }
}

Technically, outermost dictionary of declaration is as same as inner dictionaries. So you can specify '__source_root__' the dictionary.

As list

dripper can define list of dictionaries. You need to apply '__type__': 'list'.

from dripper import dripper_factory
declaration = {
    "articles": {
        "__type__": "list",
        "__source_root__": "articles",
...
        "title": "meta1",
        "author": ["meta2", "meta22"],
    }
})
d = dripper_factory(declaration)
d({
    "articles": [
        {"meta1": "Monty Python", "meta2": {"meta22": "John Doe"}},
        {"meta1": "Flying Circus", "meta2": {"meta22": "Jane Doe"}},
    ]
}) == {
    "articles": [
        {"title": "Monty Python", "author": "John Doe"},
        {"title": "Flying Circus", "author": "Jane Doe"},
    ]
}

Advanced

Converting

Use dripper.ValueDripper to pass converter function.

import dripper
declaration = {
    "title": dripper.ValueDripper(["title"], converter=lambda s: s.lower())
}
d = dripper.dripper_factory(declaration)
d({"title": "TITLE"}) == {"title": "title"}

Technically, each ends (list) will be replaced by instance of dripper.ValueDripper.

default value

Specify default keyword argument to change default value. None will be applied as default.

import dripper
declaration = {
    "title": dripper.ValueDripper(["title"], default="default")
}
d = dripper.dripper_factory(declaration)
d({}) == {"title": "default"}

Technically, each ends (list) will be replaced by instance of dripper.ValueDripper.

Combining

By combining dripper.ValueDripper, result value of that key will be combined.

import dripper
declaration = {
    "fullname": (dripper.ValueDripper(["firstname"]) +
                 dripper.ValueDripper(["lastname"]))
}
d = dripper.dripper_factory(declaration)
d({"firstname": "Hrioki", "lastname": "Kiyohara"}) == {"fullname": "HriokiKiyohara"}

CHANGES

1.2

1.1

  • None is default value of ValueDripper

    • Before this change ValueDripper without default keyword argument will raise DrippingError

    • In order to this behavior DictDripper will return empty dict when inner value dripper could not dig out values

    • Thanks for @bungoume to suggest this behavior

1.0

  • Officially supported Python 3.5

0.3.1

  • ValueDripper now accepts default argument.

0.3

  • Fixed to accept string or integer directly as source_root.

0.2

  • Improved error handling.

  • Added MixDripper.

0.1

  • Initial version

Metadata

Release files for dripper 1.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dripper 1.2
File Size Uploaded
dripper-1.2.tar.gz 5.5 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for dripper 1.2
File Interpreter ABI Platform
dripper-1.2-py3-none-any.whl Python 3 none any Details

Total release size: 12.6 kB

Release files / dripper-1.2.tar.gz

Download URL dripper-1.2.tar.gz
Size 5.5 kB
Tags Source
SHA-256 checksum
How to use checksums
f9ca9905d9997d4866064d5411d586533894515b8239f523455f20d58acfe5fb
BLAKE2b-256 checksum
How to use checksums
fd89f3436998ada1dc099d7b937f7b32e443eec4df2200f518aaf81e5f8d8a89
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Release files / dripper-1.2-py3-none-any.whl

Download URL dripper-1.2-py3-none-any.whl
Size 7.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
66385c7d1364a3fc8a8c404fca1f85760d36d27077f3dba5e05b6ed55a03a94a
BLAKE2b-256 checksum
How to use checksums
8d7ef131cec4891f718ed027dbe7677f15cab488b4d520fa77828cf3f63201c9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No

Release history Release notifications | RSS feed

This release

1.2 This release

2 release files

1.1

1 release file

1.0

1 release file

0.3.1

1 release file

0.3

1 release file

0.2

1 release file

0.1

1 release file

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page