polymo 0.9.1

Declarative REST API ingestion for PySpark

Turn REST APIs into Spark DataFrames with just a YAML file

Welcome to Polymo

Polymo makes it super easy to ingest APIs with Pyspark. You only need to define a YAML file or a Pydantic config.

My vision is that API ingestion doesn't need heavy, third party tools or hard to maintain custom code. The heck, you don't even need Pyspark skills.

How does it work?

Define a config file manually or use the recommended, lightweight builder UI. Once you are happy with your config, all you need to do is register the Polymo reader and tell Spark where to find the config:

from pyspark.sql import SparkSession
from polymo import ApiReader

spark = SparkSession.builder.getOrCreate()
spark.dataSource.register(ApiReader)

df = (
    spark.read.format("polymo")
    .option("config_path", "./config.yml")  # YAML you saved from the Builder
    .option("token", "YOUR_TOKEN")  # Only if the API needs one
    .load()
)

df.show()

Streaming works too:

spark.readStream.format("polymo")

Prefer everything in the code? Build the config with PolymoConfig model.

from pyspark.sql import SparkSession
from polymo import ApiReader, PolymoConfig

spark = SparkSession.builder.getOrCreate()
spark.dataSource.register(ApiReader)

jp_posts = PolymoConfig(
    base_url="https://jsonplaceholder.typicode.com",
    path="/posts",
)

df = (
    spark.read.format("polymo")
    .option("config_json", jp_posts.config_json())
    .load()
)
df.show()

Does it perform? Polymo can read in batches (pages in parallel) and therefore is much faster than row based solutions like UDFs.

How to start?

Locally you probably want to install polymo along with the Builder UI:

pip install "polymo[builder]"

This comes with all UI deps such as pyspark

Running Polymo on a spark cluster usually doesn't require these UI deps. In that case, just install the bare minimum deps with

pip install polymo

Launch the builder UI

polymo builder

(Optional) Run the Builder in Docker

docker compose up --build builder

• The service listens on port 8000; open http://localhost:8000 once Uvicorn reports it is running.

Where to Next

Read the docs here

Contributing

It's still early days, but Polymo already supports a lot of features! Is there something missing? Raise an issue or contribute!

Contributions and early feedback welcome!