mlslib 0.1.8

A utility library for working with data pipelines on GCP

mlslib

A lightweight utility library to simplify working with Google Cloud Storage and BigQuery on Google Cloud Platform (GCP). This library provides a set of high-level functions to streamline common data engineering and data science workflows.

🚀 Key Features

• Google Cloud Storage Integration: Upload pandas or Spark DataFrames to GCS
• File Management: Upload any local file (CSV, Parquet, Pickle, etc.) to GCS
• Public Access: Make GCS files public and get downloadable links
• BigQuery Integration: Query BigQuery tables directly into Spark DataFrames
• Notebook Display: Beautifully display PySpark DataFrames in Jupyter notebooks

📦 Installation

Install mlslib directly from PyPI:

pip install mlslib

Dependencies

The library requires the following Python packages:

• ipython>=7.0.0 - For notebook display functionality
• pyarrow>=6.0.0 - For efficient data serialization

Note: The library's functions assume that google-cloud-storage and pyspark are installed and configured in your environment.

🔧 Setup

Before using mlslib, ensure you have:

1. Google Cloud SDK installed and configured
2. Authentication set up (service account key or gcloud auth)
3. Required packages installed:

   pip install google-cloud-storage pyspark
   

📖 Usage

Google Cloud Storage Utilities (gcs_utils)

Upload Local Files to GCS

from mlslib.gcs_utils import upload_file_to_gcs

# Upload any local file to GCS
gcs_uri = upload_file_to_gcs(
    file_path="/path/to/your/local/model.pkl",
    bucket_name="my-gcp-bucket",
    gcs_path="models/model.pkl"
)
print(f"File uploaded to: {gcs_uri}")

Upload DataFrames to GCS

from mlslib.gcs_utils import upload_df_to_gcs

# Upload pandas DataFrame
gcs_path_pandas = upload_df_to_gcs(
    my_pandas_df,
    bucket_name="my-gcp-bucket",
    gcs_path="data/pandas_export.parquet",
    format="parquet"
)

# Upload Spark DataFrame
gcs_path_spark = upload_df_to_gcs(
    my_spark_df,
    bucket_name="my-gcp-bucket",
    gcs_path="data/spark_export.csv",
    format="csv"
)

Make Files Public and Get Download Links

from mlslib.gcs_utils import download_csv

# Make a GCS file public and get HTTPS download link
public_url = download_csv(
    bucket_name="my-gcp-bucket",
    file_path="data/public_file.csv"
)
print(f"Public URL: {public_url}")

BigQuery Utilities (bigquery_utils)

Load BigQuery Data into Spark DataFrames

from mlslib.bigquery_utils import load_bigquery_table_spark

# Load data from BigQuery table into Spark DataFrame
sql = "SELECT user_id, event_name FROM my_table WHERE event_date = '2025-06-22'"
df = load_bigquery_table_spark(
    spark=spark,  # Your SparkSession object
    sql_query=sql,
    table_name="my_table",
    project_id="my-gcp-project",
    dataset_id="my_analytics_dataset"
)
df.show()

Display Utilities (display_utils)

Beautiful DataFrame Display in Notebooks

from mlslib.display_utils import display_df

# Display Spark DataFrame as styled HTML table
display_df(df, limit_rows=50, title="User Events Preview")

📁 Project Structure

mlslib/
├── __init__.py          # Package initialization and exports
├── gcs_utils.py         # Google Cloud Storage utilities
├── bigquery_utils.py    # BigQuery integration utilities
└── display_utils.py     # Notebook display utilities

🔍 API Reference

gcs_utils Module

• upload_file_to_gcs(file_path, bucket_name, gcs_path) - Upload local file to GCS
• upload_df_to_gcs(df, bucket_name, gcs_path, format='parquet') - Upload DataFrame to GCS
• download_csv(bucket_name, file_path) - Make GCS file public and get download URL

bigquery_utils Module

• load_bigquery_table_spark(spark, sql_query, table_name, project_id, dataset_id) - Load BigQuery data into Spark DataFrame

display_utils Module

• display_df(df, limit_rows=100, title=None) - Display Spark DataFrame in notebook

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request. For major changes, please open an issue first to discuss what you would like to change.

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

👨‍💻 Author

Raj Jha - [e-mail]

🔗 Links

• PyPI: https://pypi.org/project/mlslib/
• Repository: https://github.com/wayfair-sandbox/dslib

📝 Changelog

Version 0.1.7

• Initial release with core GCS and BigQuery utilities
• Added DataFrame upload and display functionality
• Support for multiple file formats (CSV, Parquet, Pickle)

---

Note: This library is designed to work with Google Cloud Platform services. Make sure you have proper authentication and permissions set up before using these utilities.