specialweek 0.1.0

Declarative async pipelines with fan-out, fan-in, and built-in provenance

Special Week

Declarative async pipelines with fan-out, fan-in, and built-in provenance.

Nodes declare their own antecedents. The graph infers topology, runs everything concurrently, and attaches a full provenance chain to every result.

Installation

pip install specialweek

Demo

Fetch 20 XKCD comics, extract the first word of each title, and look every word up in a dictionary; all in parallel, declared as a single pipeline:

import asyncio
import time
from specialweek import Graph, Source, HTTP, Transform

# Fan-out: 20 items flow through the whole pipeline concurrently.
source = Source(*[{"id": str(i)} for i in range(2630, 2650)])

fetch_comic = HTTP(
    "xkcd",
    url="https://xkcd.com/${id}/info.0.json",
    extract=lambda r: {"id": r["num"], "title": r["safe_title"]},
    on_error={404: lambda _: {"id": None, "title": "(not found)"}},
    depends_on=[source],
)

first_word = Transform(
    "word",
    fn=lambda v: {"word": v["title"].split()[0].lower(), "title": v["title"], "id": v["id"]},
    depends_on=[fetch_comic],
)

define = HTTP(
    "dictionary",
    url="https://api.dictionaryapi.dev/api/v2/entries/en/${word}",
    extract=lambda r: r[0]["meanings"][0]["definitions"][0]["definition"],
    on_error={404: lambda v: f'(no definition for "{v["word"]}")'},
    concurrency=5,
    depends_on=[first_word],
)

t0 = time.perf_counter()
results = asyncio.run(Graph(define, workers=20).run())
print(f"{len(results)} comics in {time.perf_counter() - t0:.2f}s\n")

results.sort(key=lambda r: r.provenance[1]["value"]["id"] or 0)
for r in results:
    comic = r.provenance[1]["value"]
    print(f"#{comic['id']:>4}  \"{comic['title']}\"")
    print(f"        {r.value}\n")

20 comics in 0.32s

#2630  "Shuttle Skeleton"
        The part of a loom that carries the woof back and forth between the warp threads.

#2631  "Exercise Progression"
        Any activity designed to develop or hone a skill or ability.

#2632  "Greatest Scientist"
        Relatively large in scale, size, extent, number (i.e. having many parts or members) or duration (i.e. relatively long); very big.

#2633  "Astronomer Hotline"
        One who studies astronomy, the stars or the physical universe; a scientist whose area of research is astronomy or astrophysics

#2634  "Red Line Through HTTPS"
        Any of a range of colours having the longest wavelengths, 670 nm, of the visible spectrum; a primary additive colour for transmitted light: the colour obtained by subtracting green and blue from white light using magenta and yellow filters; the colour of blood, ripe strawberries, etc.

#2635  "Superintelligent AIs"
        (no definition for "superintelligent")

#2636  "What If? 2 Countdown"
        (Singlish) Used to contradict an underlying assumption held by the interlocutor.

#2637  "Roman Numerals"
        One of the main three types used for the Latin alphabet (the others being italics and blackletter), in which the ascenders are mostly straight.

#2638  "Extended NFPA Hazard Diamond"
        To increase in extent.

#2639  "Periodic Table Changes"
        Relative to a period or periods.

#2640  "The Universe by Scientific Field"
        With a comparative or with more and a verb phrase, establishes a correlation with one or more other such comparatives.

#2641  "Mouse Turbines"
        Any small rodent of the genus Mus.

#2642  "Meta-Alternating Current"
        (no definition for "meta-alternating")

#2643  "Cosmologist Gift"
        (no definition for "cosmologist")

#2644  "fMRI Billboard"
        (no definition for "fmri")

#2645  "The Best Camera"
        With a comparative or with more and a verb phrase, establishes a correlation with one or more other such comparatives.

#2646  "Minkowski Space"
        (no definition for "minkowski")

#2647  "Capri Suns"
        (no definition for "capri")

#2648  "Chemicals"
        Any specific chemical element or chemical compound or alloy.

#2649  "Physics Cost-Saving Tips"
        A medicine or drug, especially a cathartic or purgative.

Every result carries a provenance chain back to the source:

results[0].provenance
# [{"node": "source", "value": {"id": "2630"}},
#  {"node": "xkcd",   "value": {"id": 2630, "title": "Voting Software"}},
#  {"node": "word",   "value": {"word": "voting", ...}},
#  {"node": "dictionary", "value": "the action of voting"}]

Documentation

Document	Contents
Concepts	Mental model: nodes, results, fan-out, fan-in, workers, execution order
Nodes	Source, HTTP, Transform, and custom node reference
Graph	Topology discovery, the workers parameter, execution model
I/O Mapping	How values flow between nodes, key conflicts, explicit renaming
Provenance	The Result class and reading provenance chains
Examples	Five complete worked examples