TaGra 0.2.3

TaGra: TAbular data preprocessing to GRAph representation.

TaGra (Table to Graph)

TaGra is a comprehensive Python library designed to simplify the preprocessing of data, the construction of graphs from data tables, and the analysis of those graphs. It provides automated tools for handling missing data, scaling, encoding, manifold learning techniques and graph construction.

Scope of TaGra

TaGra achieves three primary objectives:

1. Automatic Data Preprocessing: TaGra automates the preprocessing of tabular data, handling missing values, scaling numeric features, encoding categorical variables and manifold learning techniques based on user-defined configurations.

2. Graph Creation: TaGra offers three distinct methods to create graphs from the data:

   • K-Nearest Neighbors (KNN): Constructs a graph by connecting each node to its k-nearest neighbors based on Euclidean distance.
   • Distance Threshold (Radius Graph): Connects nodes if the Euclidean distance between them is less than a specified threshold.
   • Similarity Graph: Adds an edge between nodes if their cosine similarity exceeds a given threshold.

When creating the graph, each row together with all its features is mapped to a node and an edge between two rows is created using the methods described above.

3. Basic Graph Analysis: TaGra provides functions to analyze the generated graphs, including degree distribution and community composition analysis.

Configuration File

The configuration file is a JSON file that contains all the settings required for preprocessing and graph creation. Below are the key settings:

• input_dataframe: Path to the input DataFrame. Supported extensions are csv, xlsx, pickle, json, parquet, hdf, h5.
• output_directory: Path to the folder where the results will be collected. If not specified, the current directory is used.
• preprocessed_filename: Filename of the preprocessed DataFrame. If not specified, a default name pattern is used.
• graph_filename: Filename of the graph file. If not specified, a default name pattern is used.
• numeric_columns: List of numeric columns.
• categorical_columns: List of categorical columns.
• target_columns: List of target columns used for graph coloring and neighborhood statistics.
• ignore_columns: List of columns to ignore during preprocessing.
• unknown_column_action: Action for unspecified columns. Options: 'infer' or 'ignore'.
• numeric_threshold: Threshold for inferring numeric columns.
• numeric_scaling: Scaling mode for numeric columns. Options: 'standard' or 'minmax'.
• categorical_encoding: Encoding for categorical columns. Options: 'one-hot' or 'label'.
• nan_action: Action for NaN values. Options: 'drop row', 'drop column', or 'infer'.
• nan_threshold: Threshold for dropping columns based on NaN ratio.
• verbose: Flag for detailed output.
• manifold_method: Method for manifold learning. Options: 'Isomap' or null.
• manifold_dimension: Number of dimensions for manifold learning output.
• method: Method to infer the graph. Options: 'knn', 'distance_threshold', or 'similarity'.
• k: Number of neighbors for 'knn' method.
• distance_threshold: Distance threshold for 'distance_threshold' method.
• similarity_threshold: Similarity threshold for 'similarity' method.
• clustering_method: Method for clustering analysis. (TODO)
• inconsistency_threshold: Threshold for inconsistency in clustering. (TODO)
• neigh_prob_path: Filename for neighborhood statistics.
• prob_heatmap_filename: Filename for heatmap of neighborhood statistics.
• degree_distribution_filename: Filename for degree distribution plot.
• betweenness_distribution_filename: Filename for betweenness centrality distribution plot.
• community_composition_filename: Filename for community composition histogram.
• graph_visualization_filename: Filename for graph visualization. If null, graph is not plotted.

Functions

Preprocessing

TaGra provides automatic data preprocessing that includes:

• Handling missing values based on user-defined settings.
• Scaling numeric features using standard or min-max scaling.
• Encoding categorical variables using one-hot or label encoding.
• Inferring the type of unspecified columns based on a threshold.

Graph Creation

TaGra supports three methods for creating graphs from preprocessed data:

1. K-Nearest Neighbors (KNN):

   • Connects each node to its k-nearest neighbors.
   • Requires the parameter k to specify the number of neighbors.

2. Distance Threshold (Radius Graph):

   • Connects nodes if their Euclidean distance is below a specified threshold.
   • Requires the parameter distance_threshold.

3. Similarity Graph:

   • Adds an edge between nodes if their cosine similarity is above a specified threshold.
   • Requires the parameter similarity_threshold.

Graph Analysis

TaGra includes basic graph analysis functions:

• Degree Distribution: Plots the degree distribution of the graph.
• Community Composition: Analyzes and plots the composition of communities within the graph.
• Neighbor class probability: Evaluates the probability of extracting a node of class $j$ in the neighborhood of a node of class $i$.

Installation

To install TaGra, simply use pip:

pip install tagra

Quickstart

python3 go.py -c examples/config.json

You can edit the option in examples/config.json and adapt them as you wish. The default option will produce a prepreocessing and a graph based on the moons dataset (SciKit Learn).

Quickerstart

python3 go.py -d path/to/dataframe -a class_name

This will preprocess, make the knn graph of path/to/dataframe. Optionally you can add the name of the target column with -a.

Usage

Settings

The settings can be specified in a configuration file. It must be a JSON file that contains the settings required for preprocessing and graph creation. Below are the key settings:

• input_dataframe: DataFrame path. Supported extensions are: csv, xlsx, pickle, json, parquet, hdf, h5. In the case of a .csv file, the presence of the header will be deduced in the preprocessing part. It is the only mandatory argument.
• output_directory: Path to the folder where the results will be collected. If not specified, the path from where the executable was launched will be used. If the folder does not exist, it will be created.
• preprocessed_filename: Filename of the preprocessed dataframe. If not specified, a name with this pattern is created: {basename}_{timestamp}.{ext} where {basename} is the name of the input_dataframe, {timestamp} is a string in the format ‘%Y%m%d%H%M’ and {ext} is the file extension. The supported extensions are the same as for input_dataframe.
• graph_filename: Filename of the graph file. If not specified, a name with the same pattern as before is created. Supported extension: .graphml.
• inferred_columns_filename: Filename for saving the inferred column types. If not specified, it will not be created. Supported extension: .pickle.
• numeric_columns: A list containing the numeric columns.
• categorical_columns: A list containing the categorical columns.
• target_columns: A list containing the "target" variable, used only to color the graph and to evaluate the statistics on the neighborhood in the resulting graph.
• ignore_columns: A list containing the columns to be ignored in the preprocessing.
• unknown_column_action: An action to deal with columns that have not been specified. Available options: 'infer' (infer how to deal with those columns) or 'ignore' (ignore the columns).
• numeric_threshold: Threshold to determine if a column is numeric when unknown_column_action is infer. If the ratio of unique instances to total rows exceeds this threshold, the column is added to numeric_columns; otherwise, to categorical_columns.
• numeric_scaling: Scaling mode for numeric_columns. Available options: 'standard' (Standard Scaler) or 'minmax' (MinMax Scaler). Notice that if a numerical columns must be ignored, it should be added to the list in ignore_columns.
• categorical_encoding: Encoding for categorical_columns. Available options: 'one-hot' (One-Hot-Encoding) or 'label' (Label Encoding).
• nan_action: An action to deal with NaN values. Options: 'drop row', 'drop column' or 'infer' (fills with the average).
• nan_threshold: If nan_action is 'drop column', the column will be dropped if the ratio of NaNs in the column to the total number of rows is greater than this value.
• verbose: A flag to print detailed output.
• manifold_method: Method for applying manifold learning on numeric_columns. Options are Isomap, TSNE, or None (to avoid manifold learning). The output dimension is always 2 and will be used to visualize the output graph.
• method: Method to infer the graph. Available options: 'knn' (make a graph with the k-nearest neighbors based on Euclidean distance), 'distance' (put an edge between nodes if their Euclidean distance is less than distance_threshold), 'similarity' (add an edge between two nodes if their cosine similarity is more than similarity_threshold).
• k: Number of neighbors if method is 'knn'.
• distance_threshold: Distance threshold; if the Euclidean distance between two rows is less than the threshold, add an edge between the rows.
• similarity_threshold: Similarity threshold; if the cosine similarity between two rows is greater than the threshold, add an edge between the rows.
• neigh_prob_path: Filename containing the statistics on the neighbors.
• degree_distribution_filename: Filename with the log-log degree distribution plot.
• community_filename: Filename with the community distribution histogram.
• graph_visualization_filename: Path to the file where the graph visualization will be saved. If null, the graph will not be plotted.
• prob_heatmap_filename: Filename of the heatmap containing the statistics on the neighbors.
• overwrite: A flag indicating whether to overwrite the results of experiments or not. If set to False, all output filenames are equipped with a timestamp, otherwise outputs are overwritten.

Data Preprocessing

from tagra.preprocessing import preprocess_dataframe

# Example usage

df_preprocessed = preprocess_dataframe(
    input_dataframe='moons.csv',
    inferred_columns_filename = 'inferred_columns_moon.pickle'
)

It will produce a preprocessed dataframe of moons.csv in the results/ directory with name moons_{timestap}.csv, where time stamp is a string in the format ‘%Y%m%d%H%M’ with the current time.

List of optional arguments and their default values

output_directory = "results/", 
preprocessed_filename = None,
inferred_columns_filename = None, 
numeric_columns = [],
categorical_columns = [], 
target_columns = [], 
unknown_column_action = 'infer',
ignore_columns = [], 
numeric_threshold = 0.05,
numeric_scaling = 'standard', 
categorical_encoding = 'one-hot',
nan_action = 'infer', 
nan_threshold = 0.5,
verbose = True, 
manifold_method = None, 
manifold_dim = None,
overwrite = False

Graph construction

from tagra.graph import construct_graph

Example usage

graph = construct_graph(
    input_dataframe='moons.csv',
    preprocessed_dataframe = df_processed # The output of preprocess_dataframe.
    inferred_columns_filename = 'inferred_columns_moon.pickle'
)

This example uses the construct_graph function to generate a graph with the distances from the preprocessed DataFrame df_preprocessed derived from 'moons.csv' and the inferred columns dictionary from inferred_columns_filename. moons.csv will be used to add the features to the nodes. The name of the graph will be graph_{timestamp}.graph. This function returns an igraph object.

List of optional arguments and their default values

preprocessed_dataframe=None,
inferred_columns_filename=None,
numeric_columns=None,
output_directory=None,
graph_filename=None,
method='knn',
k=5,
distance_threshold=None,
similarity_threshold=None,
verbose=True,
overwrite=False

Graph Analysis

Simple graph analysis.

Example usage

from tagra.analysis import analyze_graph

config = load_config(path_to_config)
if config['manifold_method'] is not None:
    pos = manifold_pos
else:
    pos = None

# Example usage
results = analyze_graph(
    graph, # The output of construct_graph.
    target_attributes='class',
    pos = pos
)

graph will be analized and the a basic analysis on the graph will be performed. pos will be used for the visualization.

List of optional arguments and their default values

target_attributes=None, 
verbose=True,
pos=None,
output_directory=None,
neigh_prob_filename = None,
degree_distribution_filename = None,
prob_heatmap_filename = None,
community_filename = None,
graph_visualization_filename = None,
overwrite = False

Contributing

We welcome contributions from the community. If you would like to contribute, please read our Contributing Guide for more information on how to get started.

License

This project is licensed under the MIT License. See the LICENSE file for more details.

Support

If you have any questions or need help, please feel free to open an issue on our GitHub repository.

Author

Davide Torre: dtorre[at]luiss[dot]it Davide Chicco: davidechicco[at]davidechicco[dot]it

---

Thank you for using TaGra! We hope it makes your data preprocessing and graph analysis tasks easier and more efficient.