Skip to main content

Spark SQL magic command for Jupyter notebooks

Project description

sparksql-magic

Spark SQL magic command for Jupyter notebooks.

Example

Prerequisites

  • Python >= 3.6
  • PySpark >= 2.3.0
  • IPython >= 7.4.0

Install

pip install sparksql-magic

Usage

Load

%load_ext sparksql_magic

Config

%config SparkSql.max_num_rows=<INT>
Option Default Description
SparkSql.max_num_rows 20 The maximum number of rows to display

Parameter

%%sparksql [-c] [-v VIEW] [variable]
<QUERY>
Parameter Description
-c --cache Cache dataframe
-e --eager Cache dataframe with eager load
-v VIEW --view VIEW Create or replace temporary view
variable Capture dataframe in a local variable

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for sparksql-magic, version 0.0.2
Filename, size File type Python version Upload date Hashes
Filename, size sparksql_magic-0.0.2-py36-none-any.whl (4.2 kB) File type Wheel Python version py36 Upload date Hashes View hashes
Filename, size sparksql-magic-0.0.2.tar.gz (3.0 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page