hadoop_mock

Utility to mock hadoop command-line interface at your home.

Overview

$ export HADOOP_SSH_HOST=rnd07.local
$ export HADOOP_SSH_OPTS=User=afyodorov
$ export HADOOP_SSH_SUDO=hdfs

$ hadoop_ssh fs -ls /flume/logs/bs/2014/09/03

Found 4 items
drwxr-xr-x   - hdfs supergroup          0 2014-10-01 16:28 /flume/logs/bs/2014/09/03/20
drwxr-xr-x   - hdfs supergroup          0 2014-10-01 16:29 /flume/logs/bs/2014/09/03/21
drwxr-xr-x   - hdfs supergroup          0 2014-10-01 16:31 /flume/logs/bs/2014/09/03/22
drwxr-xr-x   - hdfs supergroup          0 2014-10-01 16:33 /flume/logs/bs/2014/09/03/23

$ export HADOOP_MOCK_HDFS_PATH=~/.hdfs_mock
$ export HADOOP_MOCK_CLI=hadoop_ssh

$ hadoop_mock mock /flume/logs/bs/2014/09/03
$ hadoop_mock -fs ls /flume/logs/bs/2014/09/03

Found 4 items
drwxr-xr-x   - user group        204 2015-04-20 02:40 /flume/logs/bs/2014/09/03/20
drwxr-xr-x   - user group        204 2015-04-20 02:40 /flume/logs/bs/2014/09/03/21
drwxr-xr-x   - user group        204 2015-04-20 02:40 /flume/logs/bs/2014/09/03/22
drwxr-xr-x   - user group        204 2015-04-20 02:40 /flume/logs/bs/2014/09/03/23

$ seq 10 | hadoop_mock fs -put - /test.txt
$ hadoop_mock jar hadoop-streaming -reducer 'wc -l' -input /test.txt -output /output
$ cat ~/.hdfs_mock/output/*

      10

Metadata

Release files for hadoop_mock 0.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for hadoop_mock 0.1
File	Size	Uploaded
hadoop_mock-0.1.tar.gz	4.7 kB	Apr 20, 2015	Details

Built distribution (wheel)

Table of built distributions (wheels) for hadoop_mock 0.1
File	Interpreter	ABI	Platform	Reset
hadoop_mock-0.1-py2.7.egg 12.2 kB Apr 20, 2015	Legacy Egg format	-	-	Details

Total release size: 16.8 kB

Release files / hadoop_mock-0.1.tar.gz

Download URL	hadoop_mock-0.1.tar.gz
Size	4.7 kB
Tags	Source
SHA-256 checksum How to use checksums	`c894ea79ad1d4bf98a6349e697cdcd223a73d2e5d37d54c4fe5627788e8682f8`
BLAKE2b-256 checksum How to use checksums	`c87b12596045d39892a1dbf470fd2e560ba967dac07e9433fba9526a44af5a87`
Upload date	Apr 20, 2015
Uploaded using Trusted Publishing? What is trusted publishing?	No

Release files / hadoop_mock-0.1-py2.7.egg

Download URL	hadoop_mock-0.1-py2.7.egg
Size	12.2 kB
Tags	Egg
SHA-256 checksum How to use checksums	`b410548d0374ad3b3588260961db4c508c4d0d5e1c144f389d4eac2f215ae9aa`
BLAKE2b-256 checksum How to use checksums	`95b33b5abd864e34de7aed6f71098543a6635584831c3646a723e2151eeb5fbc`
Upload date	Apr 20, 2015
Uploaded using Trusted Publishing? What is trusted publishing?	No

Release history Release notifications | RSS feed

This release

0.1 This release

Apr 20, 2015 2 release files

hadoop_mock 0.1