utilsds 1.1.0

Solution for DS Team

utilsds

Utilsds is a library that includes classes and functions used in data science projects such as:

• ds_statistics:

  • test_kruskal_wallis: Perform the Kruskal-Wallis statistical test.

• transform_data:

  • DataTransformer: Transform data using various methods.

• data_processing:

  • encode_one_hot: Encode categorical features using one-hot encoding.
  • convert_numerical_to_categorized: Convert numerical features to categorized intervals.
  • scale_train_test: Scale training and testing datasets.
  • resample_X_y: Resample training data and target columns.

• data_split:

  • train_test_validation_split: Split data into training, testing, and validation sets.

• visualization:

  • MetricsPlot: Compare metrics for different parameter values.
  • Radar: Create radar plots for visualizing data.
  • cluster_characteristics: Analyze cluster characteristics.
  • comparison_density: Compare density distributions.
  • feature_distribution_box: Visualize feature distributions per cluster.
  • elbow_visualisation: Visualize the elbow method for clustering.
  • describe_clusters_metrics: Describe metrics for clusters.
  • category_null_variables: Visualize null variables in categorical data.
  • normal_distr_plots: Visualize normal distribution plots.
  • distplot_limitations: Visualize limitations of distplot.
  • boxplot_limitations: Visualize limitations of boxplot.
  • violinplot_limitations: Visualize limitations of violinplot.
  • countplot_limitations: Visualize limitations of countplot.
  • categorical_variable_perc: Visualize percentage of categorical variables.
  • spearman_correlation: Visualize spearman correlation.
  • CalculateCrammersV: Calculate Crammer's V.

• data_ops:

  • BigQuery operations:
    • load_bq_data: Load data from tables, views, and SQL files
    • save_bq_view, save_bq_table: Save views and tables
    • load_bq_procedure: Execute stored procedures
    • load_bq_details: Get table/view details and schema
    • load_bq_describe_data: Get data description using ML.DESCRIBE_DATA
    • delete_bq_data: Delete data with safety confirmations
    • dry_run: Perform dry runs to estimate query costs
  • Cloud Storage operations:
    • save_gcs_bucket: Create buckets
    • save_gcs_file, load_gcs_file: Save and load files (.pkl, .json, .csv, .html, .sql)
  • Local file operations:
    • save_local_file, load_local_file: Save and load files (.pkl, .json, .csv, .html, .sql)

• confusion_matrix:

  • ConfusionMatrix: Generate and plot confusion matrices.

• modeling:

  • Modeling: Manage modeling, metrics, and logging with Vertex AI.

• hyperopt:

  • Hyperopt: Optimize hyperparameters using Hyperopt.

• classifier:

  • Classifier: Fit, train, and manage classification models.

• experiments:

  • VertexExperiments: Manage experiments with Vertex AI.