Skip to main content

pandas API on Apache Spark
Explore Koalas docs »

Live notebook · Issues · Mailing list
Help Thirsty Koalas Devasted by Recent Fires

The Koalas project makes data scientists more productive when interacting with big data, by implementing the pandas DataFrame API on top of Apache Spark.

pandas is the de facto standard (single-node) DataFrame implementation in Python, while Spark is the de facto standard for big data processing. With this package, you can:

  • Be immediately productive with Spark, with no learning curve, if you are already familiar with pandas.
  • Have a single codebase that works both with pandas (tests, smaller datasets) and with Spark (distributed datasets).

This project is currently in beta and is rapidly evolving, with a bi-weekly release cadence. We would love to have you try it and give us feedback, through our mailing lists or GitHub issues.

Try the Koalas 10 minutes tutorial on a live Jupyter notebook here. The initial launch can take up to several minutes.

Build Status codecov Documentation Status Latest Release Conda Version Binder

Getting Started

The recommended way of installing Koalas is Conda as below.

conda install koalas -c conda-forge

You can use not only Conda but also multiple ways to install Koalas. See Installation for full instructions to install Koalas.

If you are a Databricks Runtime user, you can install Koalas using the Libraries tab on the cluster UI, or using dbutils in a notebook as below, for the regular Databricks Runtime.

dbutils.library.installPyPI("koalas")
dbutils.library.restartPython()

Note that Koalas requires Databricks Runtime 5.x or above. In the future, we will package Koalas out-of-the-box in both the regular Databricks Runtime and Databricks Runtime for Machine Learning.

Now you can turn a pandas DataFrame into a Koalas DataFrame that is API-compliant with the former:

import databricks.koalas as ks
import pandas as pd

pdf = pd.DataFrame({'x':range(3), 'y':['a','b','b'], 'z':['a','b','b']})

# Create a Koalas DataFrame from pandas DataFrame
df = ks.from_pandas(pdf)

# Rename the columns
df.columns = ['x', 'y', 'z1']

# Do some operations in place:
df['x2'] = df.x * df.x

For more details, see Getting Started and Dependencies in the official documentation.

Contributing Guide

See Contributing Guide and Design Principles in the official documentation.

FAQ

See FAQ in the official documentation.

Best Practices

See Best Practices in the official documentation.

Metadata

Release files for koalas 0.24.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for koalas 0.24.0
File Size Uploaded Provenance Actions
koalas-0.24.0.tar.gz 234.5 kB Without provenance Details for koalas-0.24.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for koalas 0.24.0
File Interpreter ABI Platform Actions
koalas-0.24.0-py3-none-any.whl Python 3 none any Details for koalas-0.24.0-py3-none-any.whl

Total release size: 780.3 kB

Release files / koalas-0.24.0.tar.gz

Download URL koalas-0.24.0.tar.gz
Size 234.5 kB
Tags Source
SHA-256 checksum
How to use checksums
1e63febea510683c47b74248ef70115fa4daa64a1700c09414e891ded247f20f
BLAKE2b-256 checksum
How to use checksums
bbcd8f09bb10eed255538e4416c9abbbf0e937f5cf8eda4a845a71d7b9315039
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.6.0 requests-toolbelt/0.9.1 tqdm/4.38.0 CPython/3.8.0

Release files / koalas-0.24.0-py3-none-any.whl

Download URL koalas-0.24.0-py3-none-any.whl
Size 545.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
e545129e941d4e7cf8f9b56a84f039f762770c3d86f459a961a438fb72cab421
BLAKE2b-256 checksum
How to use checksums
17cd2a367392c3031619b9cdbcf475ee49f3691d086f67b8545aa02dc61a0827
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.6.0 requests-toolbelt/0.9.1 tqdm/4.38.0 CPython/3.8.0

Security for koalas 0.24.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.24.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page