Skip to main content
Help us improve PyPI by participating in user testing. All experience levels needed!

Library for automatic feature extraction from JSON-datasets

Project description

Build Status Open source tool for machine learning on semi-structured data that creates numeric object-feature matrix from JSON. The idea of Datapot is to make the process of data preparation and feature extraction automatic, easy and effective.

Usage

Install Datapot:

$ git clone https://github.com/bashalex/datapot.git
$ cd datapot
$ pip install .

To create a Datapot object simply write the following:

>>> import datapot as dp
>>> data = dp.DataPot()

DataPot has two main methods:

  • fit()
  • transform()

Method fit(self, data, limit) goes through the first N objects (N = limit), passes the possible features to Transformers. Each Transformer evaluates if a feature from current field or a number of fields can be created. As a result a dict of features and Transformers is created.

To apply fit() to JSON file:

>>> f = open('data/matches_test.jsonlines', 'r')
>>> data.fit(f, limit=100)
>>> data
DataPot class instance
 - number of features without transformation: 806
 - number of new features: 315
features to transform:
    (u'players.0.gold_t', [ComplexTransformer])
    (u'picks_bans.0.is_pick', [BoolToIntTransformer])
    (u'players.0.kills_log.0.unit', [TfidfTransformer])
    (u'players.1.xp_t', [ComplexTransformer])
    (u'picks_bans.1.is_pick', [BoolToIntTransformer])
    (u'players.1.kills_log.0.unit', [TfidfTransformer])
    ...

Method transform(self, data, verbose) generates a pandas. DataFrame with new features that were detected on the fit() call. If parameter verbose is true, progress description is printed during the feature extraction.

>>> df = data.transform(f, verbose=False)
fit transformers...OK
num of new features: 315

Examples

Look for more examples of using Datapot with different datasets and more Transformer specific.

Features

Datapot provides many ways of extracting features from JSON-s.

Data types that can be processed: - Boolean - Numerical array (transform array to their sum divided by average length of array in training set) - Time series (сalculate descriptive statistical properties of a given time series) - Timestamp (date, time, day of week, day of month etc.) - Text (bag of words tf-idf, word2vec) - Categorial (one-hot encoding, dimension reduction)

Authors

  • Alex Bash
  • Yuriy Mokriy
  • Nikita Savelyev
  • Michal Rozenwald
  • Peter Romov

Datapot is a course work project of the Faculty of Computer Science of the Higher School of Economics.

Project details


Release history Release notifications

This version
History Node

0.1.3

History Node

0.1.2

History Node

0.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
datapot-0.1.3.tar.gz (15.2 kB) Copy SHA256 hash SHA256 Source None May 30, 2017

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page