streamsx.hdfs 0.1.0

IBM Streams HDFS integration

Overview

Provides functions to access files on HDFS. For example, connect to IBM Analytics Engine on IBM Cloud. This package exposes the com.ibm.streamsx.hdfs toolkit as Python methods.

• IBM Streaming Analytics

• IBM Analytics Engine

Sample

A simple hello world example of a Streams application writing string messages to a file to HDFS. Scan for created file on HDFS and read the content:

from streamsx.topology.topology import *
from streamsx.topology.schema import CommonSchema, StreamSchema
from streamsx.topology.context import submit
import streamsx.hdfs as hdfs

credentials = json.load(credentials_analytics_engine_service)

topo = Topology('HDFSHelloWorld')

to_hdfs = topo.source(['Hello', 'World!'])
to_hdfs = to_hdfs.as_string()

# Write a stream to HDFS
hdfs.write(to_hdfs, credentials=credentials, file='/sample/hw.txt')

scanned = hdfs.scan(topo, credentials=credentials, directory='/sample', init_delay=10)

# read text file line by line
r = hdfs.read(scanned, credentials=credentials)

# print each line (tuple)
r.print()

submit('STREAMING_ANALYTICS_SERVICE', topo)

Documentation

• streamsx.hdfs package documentation