An arrow interface for PySpark RDDs
Project description
Sparrow
Sparrow (a combination of Spark and arrow) is a Python mini library that enhances Spark with an arrow API.
Intent is to make mappers and filters over RDD a bit more elegant and exciting. Author also feels that here developed API does have a more consitent feel.
Consider and example of few operations on an RDD in native PySpark
...
rdd = spark.sparkContext.parallelize(
[
(1, 2.0, ["a", "b", "c"]),
(2, 3.0, ["b", "c", "d"]),
(3, 4.0, ["c", "d", "e"]),
(4, 5.0, ["d", "e", "f"]),
(5, 6.0, ["e", "f", "g"]),
]
)
res = rdd.map(lambda x: x[2]).flatMap(lambda x: x).filter(lambda x: x == 'b')
and then on RDD extended with Sparrow:
rdd = spark.sparkContext.parallelize(
[
(1, 2.0, ["a", "b", "c"]),
(2, 3.0, ["b", "c", "d"]),
(3, 4.0, ["c", "d", "e"]),
(4, 5.0, ["d", "e", "f"]),
(5, 6.0, ["e", "f", "g"]),
]
)
res = (
SparrowRDD(rdd)
>> (lambda x: x[2])
>> Flatten(lambda x: x)
>> Filter(lambda x: x == 'b')
)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pysparrow-1.0.4.tar.gz
(2.9 kB
view hashes)
Built Distribution
Close
Hashes for pysparrow-1.0.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3f7585a13190526ddaa19e81445cdc34269b458e67b6a9e4fc3f73c9ababc9ce |
|
MD5 | d8546b634d3d9913fb507ee99f9e803a |
|
BLAKE2b-256 | 882e994f001823c3aa3d319ea2b517efbeca48046457090fd32e72bec367f453 |