utilsds 2.0.0

Solution for DS Team

utilsds

Utilsds is a library that includes classes and functions used in data science projects such as:

• algorithm:

  • Algorithm: Base class for fitting, training, and getting hyperparameters of machine learning models.

• data_ops:

  • DataOperations: Handle data operations locally and with Google Cloud services (BigQuery and Cloud Storage).
  • BigQuery operations:
    • load_bq_data: Load data from tables, views, and SQL files.
    • save_bq_view, save_bq_table: Save views and tables.
    • load_bq_procedure: Execute stored procedures.
    • load_bq_details: Get table/view details and schema.
    • delete_bq_data: Delete data with safety confirmations.
    • dry_run: Perform dry runs to estimate query costs.
  • Cloud Storage operations:
    • save_gcs_bucket: Create buckets.
    • save_gcs_file, load_gcs_file: Save and load files (.pkl, .json, .csv, .html, .sql).
  • Local file operations:
    • save_local_file, load_local_file: Save and load files (.pkl, .json, .csv, .html, .sql).

• data_processing:

  • SkewnessTransformer: Transform skewed data using various methods (IHS, neglog, Yeo-Johnson, quantile).
  • NullReplacer: Replace null values in specified columns with configurable strategies.
  • ColumnDropper: Drop specified columns from a DataFrame.
  • OutliersCleaner: Clean outliers by clipping values outside specified percentile ranges.
  • CategoricalMapper: Map values in categorical columns according to a specified mapping scheme.
  • NumericalMapper: Convert numerical columns to categorical by binning.
  • Encoder: One-hot encode categorical columns in the data.
  • Normalizer: Normalize numerical columns using a provided scaler.

• data_split:

  • train_test_validation_split: Split data into training, testing, and validation sets.
  • resample_X_y: resample train data and target column.

• ds_statistics:

  • test_kruskal_wallis: Perform the Kruskal-Wallis statistical test.
  • test_agosto_pearsona: Test for normality using D'Agostino-Pearson test.

• evaluate:

  • ModelEvaluator: Evaluate models and generate plots for diagnostics.
  • ShapExplainer: Explain model predictions using SHAP values.

• experiments:

  • VertexExperiment: Manage experiments with Vertex AI.

• optuna:

  • Optuna: Optimize hyperparameters using Optuna.

• metrics:

  • Metrics: Calculate metrics for both classification and regression models.

• modeling:

  • Modeling: Manage modeling, metrics, and logging with Vertex AI.

• Supervised:

  • LazyClassifier: A classifier that automatically trains and evaluates multiple models.
  • LazyRegressor: A regressor that automatically trains and evaluates multiple models.
  • get_card_split: Function to split data into card-like groups.
  • adjusted_rsquared: Calculate adjusted R-squared for regression models.

• visualization:

  • MetricsPlot: Compare metrics for different parameter values.
  • Radar: Create radar plots for visualizing data.
  • cluster_characteristics: Analyze cluster characteristics.
  • comparison_density: Compare density distributions.
  • elbow_visualisation: Visualize the elbow method for clustering.
  • describe_clusters_metrics: Describe metrics for clusters.
  • category_null_variables: Visualize null variables in categorical data.
  • normal_distr_plots: Visualize normal distribution plots.
  • distplot_limitations: Visualize limitations of distplot.
  • boxplot_limitations: Visualize limitations of boxplot.
  • violinplot_limitations: Visualize limitations of violinplot.
  • countplot_limitations: Visualize limitations of countplot.
  • categorical_variable_perc: Visualize percentage of categorical variables.
  • spearman_correlation: Visualize spearman correlation.
  • calculate_crammers_v: Calculate Crammer's V.

• what_if_streamlit:

  • ShapSaver: Save SHAP explainer components for lazy loading in what-if analysis.
  • ColumnMetadataGenerator: Generate column metadata from a DataFrame or CSV file.

• monitoring:

  • mapping: Create column mapping from configuration file for Evidently.
  • test_data: Test data for issues using Evidently test suites.
  • check_data_drift: Check data for drift using Evidently metrics.
  • send_email_with_table: Send email notifications with HTML tables for monitoring alerts.