spark_dataframe_tools
Project description
spark_dataframe_tools
spark_dataframe_tools is a Python library that implements styles in the Dataframe
Installation
The code is packaged for PyPI, so that the installation consists in running:
pip install spark-dataframe-tools --user --upgrade
Usage
import spark_dataframe_tools
from pyspark.sql.types import StructType,StructField, StringType, IntegerType
data2 = [("James","","Smith","36636","M",3000),
("Michael","Rose","","40288","M",4000),
("Robert","","Williams","42114","M",4000),
("Maria","Anne","Jones","39192","F",4000),
("Jen","Mary","Brown","","F",-1)
]
schema = StructType([ \
StructField("firstname",StringType(),True), \
StructField("middlename",StringType(),True), \
StructField("lastname",StringType(),True), \
StructField("id", StringType(), True), \
StructField("gender", StringType(), True), \
StructField("salary", IntegerType(), True) \
])
df = spark.createDataFrame(data=data2, schema=schema)
Pandas
df_pandas = df.toPandas()
df_pandas.show2()
Spark
# Dataframe template table
df.show2()
# Dataframe memory usage
df.size()
License
New features v1.0
BugFix
- choco install visualcpp-build-tools
Reference
- Jonathan Quiza github.
- Jonathan Quiza RumiMLSpark.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Close
Hashes for spark_dataframe_tools-0.6.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8c3750822f94b2a226b3d5cb4b8cbce3886f4c78368527e2e1e89c7a97efda24 |
|
MD5 | 8292a900c66e9c2055b36ec76133c513 |
|
BLAKE2b-256 | 07b600b38d305f975437e73d13cbaff2570bb8a94d27ef50181f0a1b11ca3f92 |
Close
Hashes for spark_dataframe_tools-0.6.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eb039c2baa4a30dda84f640e7f803581371affb02a3e6e5fb20b26c94725cb17 |
|
MD5 | 2e19615047dc1a0b7c805405b4a33930 |
|
BLAKE2b-256 | 071538eb3d9a4fa44f35f0afaa74a0529a5038be00a0ce2d5507dc6405f6fbc8 |