yamconv 0.1.3

yamconv converts the file formats of machine learning datasets

yamconv

yamconv coverts a machine learning dataset from one format to another format.

Installation

yamconv is published on PyPI. You can install yamconv using pip as follows:

pip install yamconv

Alternatively, you can install it from the source code by running pip in the project directory where setup.py is located:

pip install .

Usage

yamconv.py -c converter -i input_file -o ouput_file -s settings -v

• -c: converter name
• -i: input file path
• -o: output file path
• -s: converter settings in JSON
• -v: verbose, to display the processing progress and information

Supported converters

The following are the supported converters:

• mlt.fasttext2sqlite: fastText text file to SQLite database file
• mlt.sqlite2fasttext: SQLite database file to fastText text file

Settings

Settings for converters are given in the -s option as a JSON string, e.g., '{"cache_labels": true}'.

Setting	Values	Description	Applicable converters
cache_labels	true (default), false	When cache_labels is true, the reformatting of the labels is cached in memory. It can be set to false if there is insufficient memory to cache a huge number of different labels in the dataset.	mlt.fasttext2sqlite, mlt.sqlite2fasttext

Supported dataset formats

Multi-label text classificaiton

fastText text file

The fastText format is a text file that contains a series of lines. Each line represents a text classified by multiple labels. A line starts with multiple labels, followed by the text content. Each label is marked with the __label__ prefix and the labels are separated by a space. The following is a fragment of an example fastText dataset file:

__label__food __label__region Many people love having dim sum in Hong Kong restaurants.
__label__region __label__plant __label__business The Netherlands is the major supplier to the European floral market.

SQLite database

A SQLite database is used to store the classifications of texts. The database schema is as follows:

CREATE TABLE IF NOT EXISTS texts (
    id TEXT NOT NULL PRIMARY KEY,
    text TEXT NOT NULL
);
CREATE TABLE IF NOT EXISTS labels (
    label TEXT NOT NULL,
    text_id text NOT NULL,
    FOREIGN KEY (text_id) REFERENCES texts(id)
);
CREATE INDEX IF NOT EXISTS label_index ON labels (label);
CREATE INDEX IF NOT EXISTS text_id_index ON labels (text_id);

The texts table contains the text contents in the text field, and each row is uniquely identified by the id field. The labels table contains the labels in the label field. Each row has a text_id foreign key that links the label to the text in the texts table, where the text is classified with the label. In other words, each row in texts is associated with zero or more rows in labels.

Profesional services

If you need any supporting resources or consultancy services from YAM AI Machinery, please find us at:

• https://www.yam.ai
• https://twitter.com/theYAMai
• https://www.linkedin.com/company/yamai
• https://www.facebook.com/theYAMai
• https://github.com/yam-ai
• https://hub.docker.com/u/yamai