Skip to main content

No project description provided

Project description

spark_sql_to_sqlite

Easily save a Spark dataframe into a SQLite database.

Why do you need it

Although you can use jdbc with Spark to write to SQLite, this method is error prone, and doesn't handle complex data types such as Map, Struct and Array.

spark_sql_to_sqlite automatically converts these complex data types to json, which can be used in SQLite queries using standard json functions such as json_extract.

It also has sensible defaults to create or replace the SQLite database and table(s) as needed.

How To Use

You can use this on Spark environments such as Databricks, where you have a spark variable available that represents the SparkContext.

Install it as follows:

%pip install spark-sql-to-sqlite

Then import it as follows:

from spark_sql_to_sqlite import spark_sql_to_sqlite

You can now use it as follows:

spark_sql_to_sqlite("SELECT * from {catalog}.{schema}.{table}", path_to_sqlite_db, table_name)

Example:

spark_sql_to_sqlite(spark, "SELECT * FROM hive_metastore.fire.bronze", "/tmp/fire.db", "bronze")

The above command will create a SQLite database at /tmp/fire.db if it doesn't already exists. It will create or replace the bronze table in this database and overwrite it with the results of the Spark SQL query

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

spark_sql_to_sqlite-0.8.tar.gz (2.1 kB view details)

Uploaded Source

File details

Details for the file spark_sql_to_sqlite-0.8.tar.gz.

File metadata

  • Download URL: spark_sql_to_sqlite-0.8.tar.gz
  • Upload date:
  • Size: 2.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.8.0 pkginfo/1.8.2 readme-renderer/34.0 requests/2.28.1 requests-toolbelt/0.9.1 urllib3/1.26.7 tqdm/4.62.3 importlib-metadata/4.8.1 keyring/23.1.0 rfc3986/2.0.0 colorama/0.4.4 CPython/3.9.7

File hashes

Hashes for spark_sql_to_sqlite-0.8.tar.gz
Algorithm Hash digest
SHA256 5f33fed72a03e47000ec1fdd1c3ecdbae8099505c7b1e673e1d6ac595a727af8
MD5 40d3426c054de1d3c336385470ddebce
BLAKE2b-256 2a92f1e44d2b25d3f89694d8700e1955da66d1530236300978511b4f5871b6f1

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page