polymo 0.8.1

Declarative REST API ingestion for PySpark

Welcome to Polymo

Polymo makes it super easy to ingest APIs with Pyspark. It's like slicing cake.

My vision is that API ingestion doesn't need heavy, third party tools or hard to maintain custom code. The heck, you don't even need Pyspark skills.

How does it work?

Define a config file manually or use the recommended, lightweight builder UI. Once you are happy with your config, all you need to do is register the Polymo reader and tell Spark where to find the config:

from pyspark.sql import SparkSession
from polymo import ApiReader

spark = SparkSession.builder.getOrCreate()
spark.dataSource.register(ApiReader)

df = (
    spark.read.format("polymo")
    .option("config_path", "./config.yml")  # YAML you saved from the Builder
    .option("token", "YOUR_TOKEN")  # Only if the API needs one
    .load()
)

df.show()

Structured Streaming works out of the box aswell:

stream_df = (
    spark.readStream.format("polymo")
    .option("config_path", "./config.yml")
    .option("stream_batch_size", 100)
    .option("stream_progress_path", "/tmp/polymo-progress.json")
    .load()
)

query = stream_df.writeStream.format("memory").outputMode("append").queryName("polymo")
query.show()

Does it perform? Polymo can read in batches (pages in parallel) and therefore is much faster than row based solutions like UDFs.

It's still early days, but Polymo already supports a lot of features!

• Various Authentication options
• Many Pagination patterns, plus automatic partition-aware reading when totals are exposed.
• Several partitioning stategies for parallel Spark reads.
• Incremental sync support with cursor parameters, JSON state files on local or remote storage, optional memory caching, and overrideable state keys.
• Schema controls that auto-infer types or accept Spark SQL schemas, along with record selectors, filtering expressions, and schema-based casting for nested responses.
• Structured Streaming compatibility with spark.readStream, tunable batch sizing, durable progress tracking, and a streaming smoke test mode.
• Error handling through configurable retry counts, status code lists, timeout handling, and exponential backoff settings.
• Jinja templating of query parameters gives you a ton of flexibility

How to start?

Locally you probably want to install polymo with the UI:

pip install "polymo[builder]"

This comes with UI deps such as pyspark

Running Polymo on an existing cluster in for instance databricks doesnt require these deps. In that case, just install the bare minimum depa with

pip install polymo

Launch the builder UI

polymo builder

(Optional) Run the Builder in Docker

docker compose up --build builder

• The service listens on port 8000; open http://localhost:8000 once Uvicorn reports it is running.

Where to Next

Read the docs here

Contributions and early feedback welcome!