Single pyspark API connector for all REST API's.
Welcome to Polymo
Declarative API ingestion with Pyspark. Uses the new Pyspark 4 custom data sources under the hood.
The builder generates standalone scripts
Polymo is pivoting from a runtime connector to a dev-time code generator. Fill out the Builder UI form and it exports a standalone Lakeflow Declarative Pipelines script with zero polymo runtime dependency — no config to load, nothing to import from polymo at all. The script only needs requests, the standard library, and pyspark.
For example, pointing the builder at https://jsonplaceholder.typicode.com/posts generates the following (elided below for brevity — the real file also includes retry/backoff and response-normalization helpers):
"""posts — generated by the polymo builder.
Standalone Lakeflow Declarative Pipelines source. polymo is NOT needed
at runtime; edit this file freely.
"""
BASE_URL = "https://jsonplaceholder.typicode.com"
PATH = "/posts"
PARAMS: dict = {"_limit": 20}
# … fetch/retry helpers elided …
@dp.table(name="posts")
def posts():
rows = list(fetch_records())
return spark.createDataFrame(rows)
As part of this pivot, YAML configs and the Spark custom data source described below are being phased out — they still work today but are considered legacy 0.x interfaces. See docs/superpowers/specs/2026-08-28-codegen-pivot-design.md for the full design.
How does it work?
Define a config file manually or use the recommended, lightweight builder UI. Once you are happy with your config, all you need to do is register the Polymo reader and tell Spark where to find the config:
from pyspark.sql import SparkSession
from polymo import ApiReader
spark = SparkSession.builder.getOrCreate()
spark.dataSource.register(ApiReader)
df = (
spark.read.format("polymo")
.option("config_path", "./config.yml") # YAML you saved from the Builder
.option("token", "YOUR_TOKEN") # Only if the API needs one
.load()
)
df.show()
Streaming works too:
spark.readStream.format("polymo")
Prefer everything in Python? Use the PolymoConfig model.
from pyspark.sql import SparkSession
from polymo import ApiReader, PolymoConfig
spark = SparkSession.builder.getOrCreate()
spark.dataSource.register(ApiReader)
jp_posts = PolymoConfig(
base_url="https://jsonplaceholder.typicode.com",
path="/posts",
)
df = (
spark.read.format("polymo")
.option("config_json", jp_posts.config_json())
.load()
)
df.show()
Polymo reads in batches and can read pages in parallel. Therefore Polymo can be much faster than row based solutions like UDFs.
How to start?
Locally you probably want to install polymo along with the Builder UI:
pip install "polymo[builder]"
This comes with all UI deps such as pyspark
Running Polymo on a spark cluster usually doesn't require these UI deps. In that case, just install the bare minimum deps with
pip install polymo
Launch the builder UI
polymo builder
(Optional) Run the Builder in Docker
docker compose up --build builder
- The service listens on port
8000; open http://localhost:8000 once Uvicorn reports it is running.
Where to Next
Read the docs here
Other material:
- Step by step example: medium blogpost
Contributing
It's still early days, but Polymo already supports a lot of features! Is there something missing? Raise an issue or contribute!
Contributions and early feedback welcome!
If Polymo helped, a ⭐ makes my day
1 maintainer
Metadata
1 maintainer
Release files for polymo 0.11.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| polymo-0.11.0.tar.gz | 196.8 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| polymo-0.11.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 410.9 kB
Release files / polymo-0.11.0.tar.gz
| Download URL | polymo-0.11.0.tar.gz |
|---|---|
| Size | 196.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f5246f3ef775d195f33b6bb41dca85591ea6040e3a2d686b67da54d6eaeb7f93
|
|
BLAKE2b-256 checksum How to use checksums |
fc6df35ef36c07a0c6612536892349e86688633abd5b0b2a230f37ede956cb75
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Aug 31, 2026.
Transparency logRelease files / polymo-0.11.0-py3-none-any.whl
| Download URL | polymo-0.11.0-py3-none-any.whl |
|---|---|
| Size | 214.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
184846e6117b61c0c71d0fdd16a82291af7ad6b9f823aa4e9dc0c149c6fd470f
|
|
BLAKE2b-256 checksum How to use checksums |
e150a2fa18766188efdc39be052f4e21520444b40854f30f25aae110bc0a75ca
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Aug 31, 2026.
Transparency log
