Skip to main content

Pure Python HDFS client

Project description

🐍 About This Fork🍴

This is a fork of https://github.com/spotify/snakebite, via https://github.com/kirklg/snakebite/tree/feature/python3. We maintain it enough to work for our needs at the Internet Archive. We use the library with our CDH5 cluster and have not tested it with any other versions of hadoop. Please help us improve this! Or make your own fork. No hard feelings.

Snakebite mini logo

Snakebite is a python library that provides a pure python HDFS client and a wrapper around Hadoops minicluster. The client uses protobuf for communicating with the NameNode and comes in the form of a library and a command line interface. Currently, the snakebite client supports most actions that involve the Namenode and reading data from DataNodes.

Note: all methods that read data from a data node are able to check the CRC during transfer, but this is disabled by default because of performance reasons. This is the opposite behaviour from the stock Hadoop client.

Snakebite requires python2 (python3 is not supported yet) and python-protobuf 2.4.1 or higher.

Snakebite 1.3.x has been tested mainly against Cloudera CDH4.1.3 (hadoop 2.0.0) in production. Tests pass on HortonWorks HDP 2.0.3.22-alpha (protocol versions 7 and 8)

Snakebite 2.x has been tested on Hortonworks HDP2.0 and CDH5 Beta and ONLY supports Hadoop 2.2.0 and up (protocol version 9)!

Installing

Snakebite-py3 releases will be available through pypi at https://pypi.python.org/pypi/snakebite-py3/

To install snakebite run:

pip install snakebite-py3

Documentation

More information and documentation can be found at https://snakebite.readthedocs.io/en/latest/

Development

Travis CI status: Travis Join the chat at https://gitter.im/spotify/snakebite

Copyright 2013-2016 Spotify AB Copyright 2016-2019 Internet Archive and individual contributors

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

snakebite_py3-3.0.6.tar.gz (86.7 kB view details)

Uploaded Source

Built Distribution

snakebite_py3-3.0.6-py3-none-any.whl (101.5 kB view details)

Uploaded Python 3

File details

Details for the file snakebite_py3-3.0.6.tar.gz.

File metadata

  • Download URL: snakebite_py3-3.0.6.tar.gz
  • Upload date:
  • Size: 86.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.10.12

File hashes

Hashes for snakebite_py3-3.0.6.tar.gz
Algorithm Hash digest
SHA256 f6586f79610ca466506fd8507970b3fc059b0b5144e7a8f53e9474aaed2fc637
MD5 fab55d912109fa0198285ecb65aa3148
BLAKE2b-256 98fbf206527cd961ac7f4f75bdc3e6ca7a295eee2bd134a5c982f31568a5709b

See more details on using hashes here.

File details

Details for the file snakebite_py3-3.0.6-py3-none-any.whl.

File metadata

  • Download URL: snakebite_py3-3.0.6-py3-none-any.whl
  • Upload date:
  • Size: 101.5 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.10.12

File hashes

Hashes for snakebite_py3-3.0.6-py3-none-any.whl
Algorithm Hash digest
SHA256 456ad694d0d32db24fcd23039b4fdfb1bfff168bd15b96c8bc7fabb1ac9f6d9f
MD5 97d6dacee49660f783e38f7f16f19cc3
BLAKE2b-256 a851fe16156a007d637c963a7c7dd2e3c3cf79bad0f2cfc34245cb5e156a82fe

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page