spark-dataframe-tools 0.6.14

spark_dataframe_tools

spark_dataframe_tools

spark_dataframe_tools is a Python library that implements styles in the Dataframe

Installation

The code is packaged for PyPI, so that the installation consists in running:

pip install spark-dataframe-tools --user --upgrade

Usage

import spark_dataframe_tools 

from pyspark.sql.types import StructType,StructField, StringType, IntegerType
data2 = [("James","","Smith","36636","M",3000),
    ("Michael","Rose","","40288","M",4000),
    ("Robert","","Williams","42114","M",4000),
    ("Maria","Anne","Jones","39192","F",4000),
    ("Jen","Mary","Brown","","F",-1)
  ]

schema = StructType([ \
    StructField("firstname",StringType(),True), \
    StructField("middlename",StringType(),True), \
    StructField("lastname",StringType(),True), \
    StructField("id", StringType(), True), \
    StructField("gender", StringType(), True), \
    StructField("salary", IntegerType(), True) \
  ])
 
df = spark.createDataFrame(data=data2, schema=schema)

Pandas

df_pandas = df.toPandas()
df_pandas.show2()

Spark

# Dataframe template table
df.show2()

# Dataframe memory usage
df.size()

License

Apache License 2.0.

New features v1.0

BugFix

• choco install visualcpp-build-tools

Reference

• Jonathan Quiza github.
• Jonathan Quiza RumiMLSpark.