This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Latest Version Dependencies status unknown Test status unknown Test coverage unknown
Project Description

Enhanced Python Dataframes for Spark/PySpark

Motivation

Spark DataFrames provide a nice interface to datasets that have a schema. Getting data from your code into a DataFrame in Python means creating a Row() object with field names and respective values. Given that you already have a schema with data types per field, it would be nice to easily take an object that represents the row and create the Row() object automatically.

Smartframes allow you to define a class by just creating the schema that represents the fields and datatypes. You can then create an object and set the values like any other Python class. When you are ready to store that in a DataFrame, just call the createRow() method.

The createRow() method will coerce any values into the correct data types, for example, if a field is defined as an IntegerType and the value set in the class is a String, it will attempt to convert the string to an Integer before creating the Row().

This was written when creating Row()’s with Long datatypes and finding that Spark did not handle converting integers as longs when passing values to the JVM. I needed a consistent manner to create Row() for all of my DataFrames.

Installation

pip install smartframes

Example

Simply create a class that extends from SmartFrame and define the schema as a sorted list of StructFields. It’s important that the schema is sorted as Spark gets upset if the Row() object is created with fields that are in a different order. Strange, but true.

The skipSelectedFields is a list of field names that you normally would not select when creating a select() statement.

class SimpleTable(SmartFrames):
    schema = StructType( sorted(
        [
        StructField("pk_id", IntegerType()),
        StructField("first_name", StringType()),
        ],
        key = lambda x: x.name))
    skipSelectedFields = []

...

        s1 = SimpleTable()
        s1.pk_id = 1
        s1.first_name = 'Don'

        s2 = SimpleTable()
        s2.pk_id = 2
        s2.first_name = 'Dan'
        df = self.sqlCtx.createDataFrame(self.sc.parallelize([s1.createRow(), s2.createRow()]), s1.schema)

Releases

Version Date Notes
1.1.0 10/6/2015 Performance improvements
1.0.1 10/3/2015 First release of smartframes
Release History

Release History

1.1.0

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

1.0.1

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

1.0.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

TODO: Brief introduction on what you do with files - including link to relevant help section.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
smartframes-1.1.0.tar.gz (8.3 kB) Copy SHA256 Checksum SHA256 Source Oct 7, 2015

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS HPE HPE Development Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting