Skip to main content

No project description provided

Project description

Klondike

Klondike offers a lightweight API to read and write data to Google BigQuery using Polars DataFrames.

Installation

Installing Klondike

Install at the command line

pip install klondike

Installing Rust

Since Polars leverages Rust speedups, you need to have Rust installed in your environment as well. See the Rust installation guide here.

Usage

In this demo we'll connect to Google BigQuery, read data, transform it, and write it back to the data warehouse.

First, connect to the BigQuery warehouse by supplying the BigQueryConnector() object with the relative path to your service account credentials.

from klondike import BigQueryConnector

# Connect Klondike to Google BigQuery
bq = BigQueryConnector(
    app_creds="dev/nba-player-service.json"
)

Next, supply the object with a SQL query in the read_dataframe() function to redner a DataFrame object:

# Write some valid SQL
sql = """
SELECT
    *
FROM nba_dbt.cln_player__averages
ORDER BY avg_points DESC
"""


# Pull BigQuery data into a Polars DataFrame
nyk = bq.read_dataframe(sql=sql)

Now that your data is pulled into a local instance, you can clean and transform it using standard Polars functionality - see the docs for more information.

# Perform some transformations
key_metrics = [
    "avg_offensive_rating",
    "avg_defensive_rating",
    "avg_effective_field_goal_percentage"
]

summary_stats = nyk[key_metrics].describe()

Finally, push your transformed data back to the BigQuery warehouse using the write_dataframe() function:

# Write back to BigQuery
bq.write_dataframe(
    df=summary_stats,
    table_name="nba_dbt.summary_statistics",
    if_exists="truncate"
)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

klondike-0.2.1.tar.gz (10.2 kB view details)

Uploaded Source

Built Distribution

klondike-0.2.1-py3-none-any.whl (10.8 kB view details)

Uploaded Python 3

File details

Details for the file klondike-0.2.1.tar.gz.

File metadata

  • Download URL: klondike-0.2.1.tar.gz
  • Upload date:
  • Size: 10.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.8.16

File hashes

Hashes for klondike-0.2.1.tar.gz
Algorithm Hash digest
SHA256 3d13d7166b80527691bd6581f0cecb6c04962c36e0823d236e1bade34ee3e3e5
MD5 7c13f072e65966c8f0b1faeb14bbf26a
BLAKE2b-256 c64d80f303fe7649ce46b5322d7c61742a106363863fd4d17dd1125e08d36b99

See more details on using hashes here.

File details

Details for the file klondike-0.2.1-py3-none-any.whl.

File metadata

  • Download URL: klondike-0.2.1-py3-none-any.whl
  • Upload date:
  • Size: 10.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.8.16

File hashes

Hashes for klondike-0.2.1-py3-none-any.whl
Algorithm Hash digest
SHA256 066105fd1bb5b63faae2a49ddb164f50aa7ba6296ac151186f1b87d0b12f709b
MD5 1fefe7321fd4ae0e9e1a540b507e92ad
BLAKE2b-256 f02a9994d1a0b69f07d5b82def5202f644ac0df91f22388bdb61ce077702f30c

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page