Skip to main content

A fast and memory-efficient Python feature generating framework for machine learning.

Project description

Please visit the GitHub repository for more information.

Feagen

https://img.shields.io/travis/ianlini/feagen/master.svg https://img.shields.io/pypi/v/feagen.svg https://img.shields.io/pypi/l/feagen.svg

A fast and memory-efficient Python feature generating framework for machine learning.

Introduction

TODO

Installation

pip install feagen

Getting start

Getting start from the simple lifetime prediction example is an easy way. You can first look at the raw data lifetime.csv and then you may better understand what we are doing to the data.

Creating the feature generator

The most important part in Feagen is the feature generator class. You first need to define a class like in lifetime_feature_generator.py to tell Feagen how to deal with the data. (TODO: more details)

Creating the config files

There is a command line tool feagen-init that can help create the initial config files: the global config .feagenrc/config.yml and the bundle config .feagenrc/bundle_config.yml. You can look at the comments that are automatically generated in those files or in examples/lifetime_prediction/.feagenrc to understand how to change them. (TODO: more details)

Drawing the directed acyclic graph (DAG)

There is one way for you to check if the dependency is correct. You can use the command line tool feagen-draw-dag to draw the DAG image:

usage: feagen-draw-dag [-h] [-g GLOBAL_CONFIG] [-d DAG_OUTPUT_PATH]

Generate DAG.

optional arguments:
-h, --help show this help message and exit
-g GLOBAL_CONFIG, --global-config GLOBAL_CONFIG
 the path of the path configuration YAML file (default: .feagenrc/config.yml)
-d DAG_OUTPUT_PATH, --dag-output-path DAG_OUTPUT_PATH
 output image path (default: dag.png)

You can specify the paths of the global config and the output image using -g and -d respectively. Running feagen-draw-dag -d fig/dag.png in examples/lifetime_prediction/ will give you examples/lifetime_prediction/fig/dag.png:

/examples/lifetime_prediction/fig/dag.png

(Note that the order may not be the same)

Generating features

After the generator class and the config are defined, we can now generate the features. A command line tool feagen can be used now:

usage: feagen [-h] [-g GLOBAL_CONFIG] [-b BUNDLE_CONFIG] [-d DAG_OUTPUT_PATH]
[–no-bundle]

Generate global data and data bundle.

optional arguments:
-h, --help show this help message and exit
-g GLOBAL_CONFIG, --global-config GLOBAL_CONFIG
 the path of the path configuration YAML file (default: .feagenrc/config.yml)
-b BUNDLE_CONFIG, --bundle-config BUNDLE_CONFIG
 the path of the bundle configuration YAML file (default: .feagenrc/bundle_config.yml)
-d DAG_OUTPUT_PATH, --dag-output-path DAG_OUTPUT_PATH
 draw the involved subDAG to the provided path (default: None)
--no-bundle not generate the data bundle

You can specify the paths of the global config, the bundle config, and the involved subDAG image using -g, -b and -d respectively.

The program will first find the nodes in the DAG that are involved and build a subDAG for this task, and check whether the data has been generated in the global data. The resulting DAG after these checks will be output if you specify -d. For example, in examples/lifetime_prediction/, if you run feagen first and then add a new feature height_divided_by_weight, and run feagen -d fig/involved_dag.png, you will get an image examples/lifetime_prediction/fig/involved_dag.png:

/examples/lifetime_prediction/fig/involved_dag.png

(Note that the order may not be the same)

After the subDAG is generated, the program will start running the methods you implemented in the generator class in an appropriate order, and then output to the global data. The global data will not be removed and can be reused. If you want to generate another bundle, the data that has been generated will not be generated again. This saves much time!

Finally, the data bundle is generated according to the structure specified in the bundle config. You can use hdfview to check the resulting global data and data bundle. It may help you understand what the output is. You can also use the argument --no-bundle if you don’t want to generate the data bundle (only the global data will be generated).

Now, you can use the data bundle to do machine learning!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for feagen, version 1.0.0a7
Filename, size File type Python version Upload date Hashes
Filename, size feagen-1.0.0a7-py2.py3-none-any.whl (16.5 kB) File type Wheel Python version py2.py3 Upload date Hashes View
Filename, size feagen-1.0.0a7.tar.gz (10.9 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page