Skip to main content

Provides functions for reading SequenceFile-s with Python pickles.

Project description

[![Build Status](https://travis-ci.org/src-d/sparkpickle.svg?branch=master)](https://travis-ci.org/src-d/sparkpickle) [![PyPI](https://img.shields.io/pypi/v/sparkpickle.svg)](https://pypi.python.org/pypi/sparkpickle)

SparkPickle
===========

Pure Python implementation of reading SequenceFile-s with pickles written by
Spark's [saveAsPickleFile()](http://spark.apache.org/docs/latest/api/python/pyspark.html#pyspark.RDD.saveAsPickleFile).
This is needed if you store the results from Spark in the efficient binary pickle
format and want to load them locally on your computer, without any Spark installation,
given only the actual files.

[Article about creating this project.](https://blog.sourced.tech/post/reading_pyspark_pickles_locally)

Installation
------------
```
pip install sparkpickle
```
Supports Python 2.7 and 3.x.

Usage
-----
View the contents of the file via command line:
```
python -m sparkpickle /path/to/file
```

Code:
```
import sparkpickle

for obj in sparkpickle.load_gen("/path/to/file"):
print(obj)
```

API
---
There are 3 functions: `load()`, `loads()` and `load_gen()`. The first two
are similar to those found in "pickle" package, whereas the last one is the
generator which yields deserialized objects and thus provides the minimal
memory footprint.

License
-------
Apache 2.0.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sparkpickle-1.0.1.tar.gz (19.9 kB view details)

Uploaded Source

File details

Details for the file sparkpickle-1.0.1.tar.gz.

File metadata

  • Download URL: sparkpickle-1.0.1.tar.gz
  • Upload date:
  • Size: 19.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for sparkpickle-1.0.1.tar.gz
Algorithm Hash digest
SHA256 8e60b3823a9462bb606f9b8648aa5e5101a336c2506aefffe840bc76d114cf24
MD5 c356b21876db42d082c3861df527525a
BLAKE2b-256 2e3df2af239ad6f9a1e0c851a105b663642fcbf11718aba1e014fc43d3382029

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page