Skip to main content

Spark SQL magic command for Jupyter notebooks

None

Project description

sparksql-magic

Spark SQL magic command for Jupyter notebooks.

Example

Prerequisites

  • Python >= 3.6
  • PySpark >= 2.3.0
  • IPython >= 7.4.0

Install

pip install sparksql-magic

Usage

Load

%load_ext sparksql_magic

Config

%config SparkSql.limit=<INT>
Option Default Description
SparkSql.limit 20 The maximum number of rows to display

Parameter

%%sparksql [-c|--cache] [-e|--eager] [-v|--view VIEW] [-l|--limit LIMIT] [variable]
<QUERY>
Parameter Description
-c --cache Cache dataframe
-e --eager Cache dataframe with eager load
-v VIEW --view VIEW Create or replace temporary view
-l LIMIT --limit LIMIT The maximum number of rows to display (Default: SparkSql.limit)
variable Capture dataframe in a local variable

Project details

None

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sparksql-magic-0.0.3.tar.gz (3.0 kB view hashes)

Uploaded Source

Built Distribution

sparksql_magic-0.0.3-py36-none-any.whl (4.3 kB view hashes)

Uploaded Python 3.6

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page