Python client for Hadoop® YARN API
Python client for Apache Hadoop® YARN API
Package documentation: yarn-api-client-python.readthedocs.org
REST API documentation: hadoop.apache.org
Library is compatible with Apache Hadoop 3.2.1.
If u have version other than mentioned (or vendored variant like Hortonworks), certain APIs might be not working or have differences in implementation. If u plan to use certain API long-term, you might want to make sure its not in Alpha stage in documentation.
pip install yarn-api-client
From Anaconda (conda forge)
conda install -c conda-forge yarn-api-client
From source code
git clone https://github.com/toidi/hadoop-yarn-api-python-client.git pushd hadoop-yarn-api-python-client python setup.py install popd
Enabling support for Kerberos/SPNEGO Security
- First option - using
To avoid deployment issues on a non Kerberized environment, the
dependency is optional and needs to be explicit installed in order to enable access
to YARN console protected by Kerberos/SPNEGO.
pip install requests_kerberos
From python code
from yarn_api_client.history_server import HistoryServer from requests_kerberos import HTTPKerberosAuth history_server = HistoryServer('https://127.0.0.2:5678', auth=HTTPKerberosAuth())
PS: You need to get valid kerberos ticket in systemwide kerberos cache before running your code, otherwise calls to kerberized environment won't go through (run kinit before proceeding to run code)
- Second option - using
If you want to avoid using terminal calls, you have to perform SPNEGO handshake to retrieve ticket yourself. Full API documentation: https://pythongssapi.github.io/python-gssapi/latest/
- First way
- Alternative way
python -m yarn_api_client --help
from yarn_api_client import ApplicationMaster, HistoryServer, NodeManager, ResourceManager am = ApplicationMaster('https://127.0.0.2:5678') app_information = am.application_information('application_id')
- Add support for Python 3.8.x
- Fix HTTPS url parsing
- Fix JSON body request APIs
- Handle YARN response with empty contents
- Better logging support
- Passes the authorization instance to the Active RM check
- Establishes a new (working) documentation site in readthedocs.io: yarn-api-client-python.readthedocs.io
- Adds more python version (3.7 and 3.8) to test matrix and removes 2.6.
- Major cleanup of API.
- Address/port parameters have been replaced with complete endpoints (includes scheme [e.g., http or https]).
- ResourceManager has been updated to take a list of endpoints for improved HA support.
- ResourceManager, ApplicationMaster, HistoryServer and NodeManager have been updated with methods corresponding to the latest REST API.
- pytest support on Windows has been provided.
- Documentation has been updated.
NOTE: Applications using APIs relative to releases prior to 1.0 should pin their dependency on yarn-api-client to less than 1.0 and are encouraged to update to 1.0 as soon as possible.
- Honor configured HTTP Policy when no address is provided - enabling using of HTTPS in these cases.
- Extend ResourceManager to allow applications to determine resource availability prior to submission.
- Hotfix release to fix internal signature mismatch
- More flexible support for discovering Hadoop configuration including multiple Resource Managers when HA is configured
- Properly support YARN post response codes
- Properly set Content-Type in PUT requests
- Check for HADOOP_CONF_DIR env variable
- Make Kerberos/SPNEGO dependency optional
- Fix cluster_application_kill API
- Add support for YARN endpoints protected by Kerberos/SPNEGO
- Moved to
requestspackage for REST API invocation
http_conproperty, as connections are now managed by
- Fixed History REST API
- Added compatibility with HA enabled Resource Manager
YARN API client is developed by an open community, and the current maintainers are listed below in alphabetical order:
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size yarn_api_client-1.0.2-py2.py3-none-any.whl (23.2 kB)||File type Wheel||Python version py2.py3||Upload date||Hashes View|
|Filename, size yarn-api-client-1.0.2.tar.gz (20.6 kB)||File type Source||Python version None||Upload date||Hashes View|
Hashes for yarn_api_client-1.0.2-py2.py3-none-any.whl