Skip to main content

Spark SQL magic command for Jupyter notebooks

Project description

sparksql-magic

Spark SQL magic command for Jupyter notebooks.

Example

Prerequisites

  • Python >= 3.6
  • PySpark >= 2.3.0
  • IPython >= 7.4.0

Install

pip install sparksql-magic

Usage

Load

%load_ext sparksql_magic

Config

%config SparkSql.max_num_rows=<INT>
Option Default Description
SparkSql.max_num_rows 20 The maximum number of rows to display

Parameter

%%sparksql [-c] [-v VIEW] [variable]
<QUERY>
Parameter Description
-c --cache Cache dataframe
-v VIEW --view VIEW Create or replace temporary view
variable Capture dataframe in a local variable

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sparksql-magic-0.0.1.tar.gz (2.9 kB view hashes)

Uploaded source

Built Distribution

sparksql_magic-0.0.1-py36-none-any.whl (4.1 kB view hashes)

Uploaded py36

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page