Single pyspark API connector for all REST API's.
Welcome to Polymo
Declarative API ingestion with Pyspark. Uses the new Pyspark 4 custom data sources under the hood.
How does it work?
Define a config file manually or use the recommended, lightweight builder UI. Once you are happy with your config, all you need to do is register the Polymo reader and tell Spark where to find the config:
from pyspark.sql import SparkSession
from polymo import ApiReader
spark = SparkSession.builder.getOrCreate()
spark.dataSource.register(ApiReader)
df = (
spark.read.format("polymo")
.option("config_path", "./config.yml") # YAML you saved from the Builder
.option("token", "YOUR_TOKEN") # Only if the API needs one
.load()
)
df.show()
Streaming works too:
spark.readStream.format("polymo")
Prefer everything in Python? Use the PolymoConfig model.
from pyspark.sql import SparkSession
from polymo import ApiReader, PolymoConfig
spark = SparkSession.builder.getOrCreate()
spark.dataSource.register(ApiReader)
jp_posts = PolymoConfig(
base_url="https://jsonplaceholder.typicode.com",
path="/posts",
)
df = (
spark.read.format("polymo")
.option("config_json", jp_posts.config_json())
.load()
)
df.show()
Polymo reads in batches and can read pages in parallel. Therefore Polymo can be much faster than row based solutions like UDFs.
How to start?
Locally you probably want to install polymo along with the Builder UI:
pip install "polymo[builder]"
This comes with all UI deps such as pyspark
Running Polymo on a spark cluster usually doesn't require these UI deps. In that case, just install the bare minimum deps with
pip install polymo
Launch the builder UI
polymo builder
(Optional) Run the Builder in Docker
docker compose up --build builder
- The service listens on port
8000; open http://localhost:8000 once Uvicorn reports it is running.
Where to Next
Read the docs here
Other material:
- Step by step example: medium blogpost
Contributing
It's still early days, but Polymo already supports a lot of features! Is there something missing? Raise an issue or contribute!
Contributions and early feedback welcome!
If Polymo helped, a ⭐ makes my day
1 maintainer
Metadata
1 maintainer
Release files for polymo 0.10.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| polymo-0.10.0.tar.gz | 204.0 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| polymo-0.10.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 421.6 kB
Release files / polymo-0.10.0.tar.gz
| Download URL | polymo-0.10.0.tar.gz |
|---|---|
| Size | 204.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
94159c59ee731b24a991447922e7bc49899acd3c9c44361c25bdf761355d9269
|
|
BLAKE2b-256 checksum How to use checksums |
3f1fd801b50641b68b93238576a14a07bb6a1b0e115e6b5f393dbe27b1a68ded
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Aug 27, 2026.
Transparency logRelease files / polymo-0.10.0-py3-none-any.whl
| Download URL | polymo-0.10.0-py3-none-any.whl |
|---|---|
| Size | 217.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
c4e77e03c0216c03a60c6c2380bfe666ebeb0d2b1589f1543ac9454edb3b6e40
|
|
BLAKE2b-256 checksum How to use checksums |
f1e6776ea8beb19e3604aff802dc52f5e9b291202fc0a5e919919be5b08b8c54
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Aug 27, 2026.
Transparency log
