Skip to main content

Python wrapper for YARN Applications

Project description

Build Status Coverage Status

The knit library provides a Python interface to Scala for interacting with the YARN resource manager.

View the documentation for knit.

Overview

knit allows you to use python in conjunction with YARN, the most common resource manager for Hadoop systems. It provides to following high-level entry-points:

  • CondaCreator, a way to create zipped conda environments, so that they can be uploaded to HDFS and extracted for use in YARN containers

  • YARNAPI, an interface to the YARN resource manager to get application/container statuses, logs, and to kill running jobs

  • Knit, a YARN application runner, which generates an instance of a scala-based YARN client, and launches an application on YARN, which in turn runs commands in YARN containers

  • DaskYARNCluster, launches a Dask distributed cluster on YARN, one worker process per container.

The intent is to use knit from a cluster edge-node, i.e., with YARN configuration and the CLI available locally.

Quickstart

Install from conda-forge

> conda install -c conda-forge knit

or with pip

> pip install knit

If installing from source, you must first build the java library (requires java and maven)

> python setup.py install mvn

To run an arbitrary command on the yarn cluster

import knit
k = knit.Knit()
k.start('env')  # wait some time
k.logs()

To start a dask cluster on YARN

import dask_yarn
cluster = dask_yarn.DaskYARNCluster()
cluster.start(nworkers=4, memory=1024, cpus=2)

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

knit-0.2.4.tar.gz (23.2 MB view details)

Uploaded Source

Built Distribution

knit-0.2.4-py2.py3-none-any.whl (23.2 MB view details)

Uploaded Python 2 Python 3

File details

Details for the file knit-0.2.4.tar.gz.

File metadata

  • Download URL: knit-0.2.4.tar.gz
  • Upload date:
  • Size: 23.2 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for knit-0.2.4.tar.gz
Algorithm Hash digest
SHA256 f94193d93bc785c2603c110cde98a2bb5c89b9124d6d9e9bf0cd9be680b9bbc2
MD5 91ed9f650125a7232b38c57b99c251af
BLAKE2b-256 a0d7304fbb4d32bc317a7b05e43685d7542df8cf3fa719e6db16f0b30b0144d0

See more details on using hashes here.

File details

Details for the file knit-0.2.4-py2.py3-none-any.whl.

File metadata

File hashes

Hashes for knit-0.2.4-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 7555ca1750844be48ef3352a68189911a17c33aa8000f0d9355fa6fa22e27db6
MD5 ab138a94ca1171ccbd085e994031c2aa
BLAKE2b-256 1aab88c2cfebd4e11f8df3801ad03832062cc6f6f5ed6606b5d2962b5eacbe9b

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page