A Python client for Apache Livy
Project description
Livy is an open source REST interface for interacting with Spark. pylivy is a Python client for Livy, enabling easy remote code execution on a Spark cluster.
Usage
from livy import LivySession
LIVY_URL = 'http://spark.example.com:8998'
with LivySession(LIVY_URL) as session:
# Run some code on the remote cluster
session.run("filtered = df.filter(df.name == 'Bob')")
# Retrieve the result
local_df = session.read('filtered')
You can also use pylivy asynchronously with asyncio:
import asyncio
from livy import AsyncLivySession
LIVY_URL = 'http://spark.example.com:8998'
await def do_task():
async with AsyncLivySession(LIVY_URL) as session:
# Run some code on the remote cluster
await session.run("filtered = df.filter(df.name == 'Bob')")
# Retrieve the result
local_df = await session.read('filtered')
loop = asyncio.get_event_loop()
loop.run_until_complete(do_task())
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
livy-0.2.1.tar.gz
(6.6 kB
view hashes)