webpath 0.0.1

Pathlib-like URLs with first-class Requests + optional async

Webpath

HTTP library that makes APIs actually enjoyable to work with

• JSON shortcuts - find(), find_all(), extract() with dot notation
• Request logging - Auto request/response logging
• Caching - Filters some sensitive headers automatically
• Pagination - Cycle detection and next page finding
• Rate Limiting - Auto-throttle requests

Why webpath?

vs requests/httpx:

• Manual JSON navigation: resp.json()["data"]["users"][0]["name"]
• Intuitive shortcuts: resp.find("data.users.0.name")

vs other URL builders:

• No HTTP client integration
• Easier URL building + requests + JSON processing

vs debugging tools:

• Separate tools for logging/inspection
• Built-in logging, response inspection, cURL generation

Installation

# core features
pip install webpath

# async helpers (adds httpx)
pip install "webpath[async]"

# progress bar downloads (adds tqdm)
pip install "webpath[progress]"

Quick Start

Python API

1. Build URLs like pathlib.Path

base  = WebPath("https://api.example.com/v1")
user  = base / "users" / 42
print(user) # https://api.example.com/v1/users/42

2. Add query params

detail = user.with_query(fields=["name", "email"])
print(detail) 
# …/42?fields=name&fields=email

3. One liner GET

resp = detail.get(timeout=5).json()
print(resp["name"])

4. Auto retries + back off

html = WebPath("https://httpbin.org/status/503").get(retries=3, backoff=0.5).text

5. Reuse a single session

with detail.session() as call:
    a = call("get").json()
    b = call("post", json={"hello": "world"}).json()

6. Async request (requires webpath[async])

import asyncio

async def main():
    data = await detail.aget(timeout=5)
    print(data.json())

asyncio.run(main())

7. Download with progress + checksum

url  = WebPath("https://speed.hetzner.de/100MB.bin")
path = url.download("100MB.bin", progress=True,
                    checksum="5be551ef1ce3…")
print("saved to", path)

8. Respect API rate limits

api = WebPath("https://api.github.com").with_rate_limit(2.0)  # 2 requests/sec
for user in ["octocat", "torvalds"]:
    resp = api / "users" / user  # auto waits between calls
    print(f"{resp.get().json()['name']}")

9. Debug requests in real-time

api = WebPath("https://api.github.com").with_logging()
resp = (api / "users" / "octocat").get()

# deep dive into the response
resp.inspect()

NOTE:

• with_logging() = see timing for every request (example 9)
• inspect() = deep dive into a specific response (example 9)

CLI Usage

Join path segments

webpath join https://api.example.com/v1 users 42
#### https://api.example.com/v1/users/42

simple GET (stdout)

webpath get https://api.github.com/repos/python/cpython -p | jq '.stargazers_count'

GET with retry/back off

webpath get https://httpbin.org/status/503 --retries 3 --backoff 0.5

download with retries, back off and checksum

webpath download https://speed.hetzner.de/100MB.bin 100MB.bin \
                 --retries 4 --backoff 0.5 \
                 --checksum blahblahblah…

Flags for cli

Options:

Flag	Description	Default
-p, --pretty	Pretty-print JSON responses	off
-r, --retries	Number of retry attempts	0
-b, --backoff	Back off factor in seconds	0.3

WebPath Tutorial

Step 1 - CoinGecko’s API

CoinGecko exposes an unauthenticated REST API at
https://api.coingecko.com/api/v3/.

Most market endpoints live under /coins/markets.
Example (unfriendly style):

GET https://api.coingecko.com/api/v3/coins/markets
     ?vs_currency=usd
     &ids=bitcoin,ethereum,dogecoin
     &order=market_cap_desc
     &per_page=3
     &page=1
     &sparkline=false

---

Step 2 - Step by step with WebPath

from webpath import WebPath

# create a base path (it behaves like pathlib.Path)
api = WebPath("https://api.coingecko.com/api/v3")

markets = api / "coins" / "markets"

# attach query parameters in plain Python
coins   = ["bitcoin", "ethereum", "dogecoin", "matic-network"]
params  = dict(
    vs_currency = "usd",
    ids         = ",".join(coins),
    order       = "market_cap_desc",
    per_page    = len(coins),
    sparkline   = "false"
)
url = markets.with_query(**params)

print(url)
# -> https://api.coingecko.com/api/v3/coins/markets?vs_currency=usd&ids=bitcoin,...

Quick sanity check

print(url.curl())
url.inspect()

### What it looks like

┌──────── Response ────────┐
│ 200 OK * 147 ms * 3 KB   │
└──────────────────────────┘
┌──────── Response Body ───┐
{                          |
  "id": "bitcoin",         |
  "name": "Bitcoin",       |
  …                        |
└──────────────────────────┘
┌──────── Headers ─────────┐
content-type  application/json
…
└──────────────────────────┘
</details>

---

Step 3 - Fetching & Caching

data = url.with_cache(ttl=120).get().json()

# .. second call < 2 min later is served instantly from disk

The cache is stored in ~/.webpath/cache by default and strips auth headers automatically.

---

Step 4 - Extraction

Each coin object looks like (truncated):

{
  "id": "bitcoin",
  "symbol": "btc",
  "name": "Bitcoin",
  "current_price": 68123,
  "price_change_percentage_24h": -2.17,
  
}

You can use normal indexing or WebPath’s helpers:

rows = []
for coin in data:
    r = WebPath.from_json(coin)
    rows.append((
        r["name"],
        f"${r['current_price']:,}",
        f"{r.find('price_change_percentage_24h'):+.2f} %"
    ))

---

Step 5 - Print a Table (optional)

from tabulate import tabulate
from rich.console import Console

console = Console()

rows.sort(key=lambda x: float(x[2].replace('%', '')), reverse=True)

console.print(
    tabulate(rows, headers=["Coin", "Price (USD)", "24 h %"], tablefmt="rounded_outline")
)

## Sample output**

╭────────┬──────────────┬─────────╮
│ Coin   │ Price (USD)  │ 24 h %  │
├────────┼──────────────┼─────────┤
│ Dogecoin │ $0.245     │ +3.18 % │
│ Bitcoin  │ $68,123    │ -2.17 % │
│ …        │ …          │ …       │
╰────────┴──────────────┴─────────╯

---

Step 6 - Making It Reusable

Create a tiny helper module crypto.py:

from webpath import WebPath

API = WebPath("https://api.coingecko.com/api/v3") / "coins" / "markets"

def get_market(coins, currency="usd", ttl=120):
    url = API.with_query(
        vs_currency=currency,
        ids=",".join(coins),
        order="market_cap_desc",
        per_page=len(coins),
        sparkline="false"
    ).with_cache(ttl=ttl)
    return url.get().json()

Now in any script / notebook:

from crypto import get_market
print(get_market(["solana", "toncoin"], "eur")[0]["current_price"])

---

Step 7 - Going Further

1. Pagination – CoinGecko allows page=; chain .paginate() to stream everything
2. Async – replace .get() with .aget() inside get_market after
   pip install "webpath[async]"
3. Retries/Back off – url.get(retries=3, backoff=0.5) for flaky networks
4. Downloads – Need historical csv dumps? Use .download() with checksums
5. CLI – webpath get "$(crypto_url)" -p to poke around quickly