Skip to main content

Map, Clean, Merge. For mapping data from one schema to another.

Project description


Core MCM - Map Clean Merge


MCM has two main peices, a reader and a mapper.

- Reader
* Reads csv files, returns a generator of DictCSVReader parsed rows.
* Optionally chunks the rows into groupings of specified sizes.
- Mapper
* Can build a probabalistic column mapping given a schema and some raw data.
* Will substitute saved values for suggested mapping (e.g. pulling a previous mapping from DB).
* Totally flexible, you pass a callable which takes the raw data and returns a mapping.
* Will clean data based on a Cleaner object for a given type. Type is inferred from the mapping schema.
* Ability to set "initial_data"
* If you always need to set some information in the object that you're mapping data into, this is useful.
* Concatenate rows together with a specified delimiter character.
* Data which doesn't match a given schema's mapping is still saved. It's put in a dictionary called ``extra_data``.


Once it's hosted on Pypi:
pip install mcm-core


from mcm import cleaners, mapper, reader

# Here our mapping is just a dictionary where our keys are raw data representations
# and our values are our normalized attributes that we're mapping to.
mapping = {'Thing': 'thing_1', 'Other thing': 'thing_2'}

# model_class can be any type of object.
model_class = object

# Reading and mapping from a CSV file, simple case.
parser = reader.MCMParser(csv_file_handle)
mapped_objs = [m for m in parser.map_rows(mapping, model_class)]


1. Clone.
2. Create a virtualenv; if you use virtualenv wrapper you'll need to
1. Run ``python develop`` to hardlink your files into your env.


Unfortunately, there are some directory path issues still baked in.
To run tests you have to be in the ``tests`` directory:

cd tests && nosetests

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for mcm, version 0.0.1
Filename, size File type Python version Upload date Hashes
Filename, size mcm-0.0.1.tar.gz (27.5 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page