pandas API on Apache Spark
Explore Koalas docs »
Live notebook
·
Issues
·
Mailing list
Help Thirsty Koalas Devasted by Recent Fires
The Koalas project makes data scientists more productive when interacting with big data, by implementing the pandas DataFrame API on top of Apache Spark.
pandas is the de facto standard (single-node) DataFrame implementation in Python, while Spark is the de facto standard for big data processing. With this package, you can:
- Be immediately productive with Spark, with no learning curve, if you are already familiar with pandas.
- Have a single codebase that works both with pandas (tests, smaller datasets) and with Spark (distributed datasets).
This project is currently in beta and is rapidly evolving, with a bi-weekly release cadence. We would love to have you try it and give us feedback, through our mailing lists or GitHub issues.
Try the Koalas 10 minutes tutorial on a live Jupyter notebook here. The initial launch can take up to several minutes.
Getting Started
The recommended way of installing Koalas is Conda as below.
conda install koalas -c conda-forge
You can use not only Conda but also multiple ways to install Koalas. See Installation for full instructions to install Koalas.
If you are a Databricks Runtime user, you can install Koalas using the Libraries tab on the cluster UI, or using dbutils in a notebook as below, for the regular Databricks Runtime.
dbutils.library.installPyPI("koalas")
dbutils.library.restartPython()
Note that Koalas requires Databricks Runtime 5.x or above. In the future, we will package Koalas out-of-the-box in both the regular Databricks Runtime and Databricks Runtime for Machine Learning.
Now you can turn a pandas DataFrame into a Koalas DataFrame that is API-compliant with the former:
import databricks.koalas as ks
import pandas as pd
pdf = pd.DataFrame({'x':range(3), 'y':['a','b','b'], 'z':['a','b','b']})
# Create a Koalas DataFrame from pandas DataFrame
df = ks.from_pandas(pdf)
# Rename the columns
df.columns = ['x', 'y', 'z1']
# Do some operations in place:
df['x2'] = df.x * df.x
For more details, see Getting Started and Dependencies in the official documentation.
Contributing Guide
See Contributing Guide and Design Principles in the official documentation.
FAQ
See FAQ in the official documentation.
Best Practices
See Best Practices in the official documentation.
Metadata
Release files for koalas 0.24.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| koalas-0.24.0.tar.gz | 234.5 kB | Without provenance | Details for koalas-0.24.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| koalas-0.24.0-py3-none-any.whl | Python 3 | none | any | Details for koalas-0.24.0-py3-none-any.whl |
Total release size: 780.3 kB
Release files / koalas-0.24.0.tar.gz
| Download URL | koalas-0.24.0.tar.gz |
|---|---|
| Size | 234.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1e63febea510683c47b74248ef70115fa4daa64a1700c09414e891ded247f20f
|
|
BLAKE2b-256 checksum How to use checksums |
bbcd8f09bb10eed255538e4416c9abbbf0e937f5cf8eda4a845a71d7b9315039
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.6.0 requests-toolbelt/0.9.1 tqdm/4.38.0 CPython/3.8.0
|
Release files / koalas-0.24.0-py3-none-any.whl
| Download URL | koalas-0.24.0-py3-none-any.whl |
|---|---|
| Size | 545.7 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
e545129e941d4e7cf8f9b56a84f039f762770c3d86f459a961a438fb72cab421
|
|
BLAKE2b-256 checksum How to use checksums |
17cd2a367392c3031619b9cdbcf475ee49f3691d086f67b8545aa02dc61a0827
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/2.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.6.0 requests-toolbelt/0.9.1 tqdm/4.38.0 CPython/3.8.0
|
Security for koalas 0.24.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.