Sparkly
Helpers & syntax sugar for PySpark. There are several features to make your life easier:
Definition of spark packages, external jars, UDFs and spark options within your code;
Simplified reader/writer api for Cassandra, Elastic, MySQL, Kafka;
Testing framework for spark applications.
More details could be found in the official documentation.
Installation
Sparkly itself is easy to install:
pip install sparkly
The tricky part is pyspark. There is no official distribution on PyPI. As a workaround we can suggest:
Use env variable PYTHONPATH to point to your Spark installation, something like:
export PYTHONPATH="/usr/local/spark/python/lib/pyspark.zip:/usr/local/spark/python/lib/py4j-0.10.4-src.zip"
Use our setup.py file for pyspark. Just add this to your requirements.txt:
-e git+https://github.com/Tubular/spark@branch-2.1.0#egg=pyspark&subdirectory=python
Here in Tubular, we published pyspark to our internal PyPi repository.
Getting Started
Here is a small code snippet to show how to easily read Cassandra table and write its content to ElasticSearch index:
from sparkly import SparklySession
class MySession(SparklySession):
packages = [
'datastax:spark-cassandra-connector:2.0.0-M2-s_2.11',
'org.elasticsearch:elasticsearch-spark-20_2.11:6.5.4',
]
if __name__ == '__main__':
spark = MySession()
df = spark.read_ext.cassandra('localhost', 'my_keyspace', 'my_table')
df.write_ext.elastic('localhost', 'my_index', 'my_type')
See the online documentation for more details.
Testing
To run tests you have to have docker and docker-compose installed on your system. If you are working on MacOS we highly recommend you to use docker-machine. As soon as the tools mentioned above have been installed, all you need is to run:
make test
Supported Spark Versions
At the moment we support:
sparkly >= 2.7 | Spark 2.4.x |
sparkly 2.x | Spark 2.0.x and Spark 2.1.x and Spark 2.2.x |
sparkly 1.x | Spark 1.6.x |
3 maintainers
Metadata
3 maintainers
Release files for sparkly 2.8.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| sparkly-2.8.2.tar.gz | 33.7 kB | Without provenance | Details |
Release files / sparkly-2.8.2.tar.gz
| Download URL | sparkly-2.8.2.tar.gz |
|---|---|
| Size | 33.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
6b20381e01718dc2a783e67ec1d392ad6caa2437f028f12bbf97823c653eb948
|
|
BLAKE2b-256 checksum How to use checksums |
fd8b3a8e8390ce158be3f2a7b0e6fa7fcb6f6db837e6fe687e1181102196b7db
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
Python-urllib/3.7
|
Security for sparkly 2.8.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
