hdfs 2.0.0

HdfsCLI: API and command line interface for HDFS.

API and command line interface for HDFS.

$ hdfscli --alias=dev

Welcome to the interactive HDFS python shell.
The HDFS client is available as `CLIENT`.

In [1]: CLIENT.list('models/')
Out[1]: ['1.json', '2.json']

In [2]: with CLIENT.read('models/2.json') as reader:
  ...:     from json import load
  ...:     model = load(reader)
  ...:     model['normalize'] = False
  ...:

In [3]: with CLIENT.write('models/2.json', overwrite=True) as writer:
  ...:     from json import dump
  ...:     dump(model, writer)
  ...:

Features

• Python (2 and 3) bindings for the WebHDFS (and HttpFS) API, supporting both secure and insecure clusters.

• Command line interface to transfer files and start an interactive client shell, with aliases for convenient namenode URL caching.

• Additional functionality through optional extensions:

  • kerberos, adding support for Kerberos authenticated clusters.

  • avro, allowing reading and writing Avro files directly from HDFS.

  • dataframe, enabling fast loading and saving of pandas dataframes on HDFS.

See the documentation to learn more.

Testing

HdfsCLI is tested against both WebHDFS and HttpFS. There are two ways of running tests (see scripts/ for helpers to set up a test HDFS cluster):

$ HDFSCLI_TEST_URL=http://localhost:50070 nosetests # Using a namenode's URL.
$ HDFSCLI_TEST_ALIAS=dev nosetests # Using an alias.

Contributing

We’d love to hear what you think on the issues page. Pull requests are also most welcome!