livy 0.2.0

A Python client for Apache Livy

Livy is an open source REST interface for interacting with Spark. pylivy is a Python client for Livy, enabling easy remote code execution on a Spark cluster.

Usage

from livy import LivySession

LIVY_URL = 'http://spark.example.com:8998'

with LivySession(LIVY_URL) as session:
    # Run some code on the remote cluster
    session.run("filtered = df.filter(df.name == 'Bob')")
    # Retrieve the result
    local_df = session.read('filtered')

You can also use pylivy asynchronously with asyncio:

import asyncio
from livy import AsyncLivySession

LIVY_URL = 'http://spark.example.com:8998'

await def do_task():

    async with AsyncLivySession(LIVY_URL) as session:
        # Run some code on the remote cluster
        await session.run("filtered = df.filter(df.name == 'Bob')")
        # Retrieve the result
        local_df = await session.read('filtered')

loop = asyncio.get_event_loop()
loop.run_until_complete(do_task())