IBM Streams HDFS integration
Project description
Overview
Provides functions to access files on HDFS. For example, connect to IBM Analytics Engine on IBM Cloud. This package exposes the com.ibm.streamsx.hdfs toolkit as Python methods.
Sample
A simple hello world example of a Streams application writing string messages to a file to HDFS. Scan for created file on HDFS and read the content:
from streamsx.topology.topology import * from streamsx.topology.schema import CommonSchema, StreamSchema from streamsx.topology.context import submit import streamsx.hdfs as hdfs credentials = json.load(credentials_analytics_engine_service) topo = Topology('HDFSHelloWorld') to_hdfs = topo.source(['Hello', 'World!']) to_hdfs = to_hdfs.as_string() # Write a stream to HDFS hdfs.write(to_hdfs, credentials=credentials, file='/sample/hw.txt') scanned = hdfs.scan(topo, credentials=credentials, directory='/sample', init_delay=10) # read text file line by line r = hdfs.read(scanned, credentials=credentials) # print each line (tuple) r.print() submit('STREAMING_ANALYTICS_SERVICE', topo)
Documentation
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
streamsx.hdfs-1.0.0.tar.gz
(5.5 kB
view hashes)
Built Distribution
Close
Hashes for streamsx.hdfs-1.0.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d2f5ef1344853bdf34308f7249d31afedaad91031a2b670de33a0e9861db687b |
|
MD5 | 5ba73976535cb2b4c2df43ea864dffa8 |
|
BLAKE2b-256 | 4864a033f49d365ab4ca62b10b45cc11329b9226ee3651926b298ea2179bcc8d |