A secure curl wrapper with middleware support and HTML-to-markdown extraction
Project description
scurl
A secure curl wrapper with middleware support and HTML-to-markdown extraction.
Installation
pip install sibylline-scurl
Or with pipx (recommended for CLI tools):
pipx install sibylline-scurl
Usage
# Fetch a URL and extract clean markdown from HTML
scurl https://example.com
# Raw output (disable response middleware)
scurl --raw https://example.com
# All curl flags work
scurl -H "Accept: application/json" https://api.example.com/data
Features
- SecretDefender: Automatically detects and blocks requests containing exposed secrets/tokens
- ReadabilityExtractor: Extracts clean markdown from HTML responses using readability + html2text
- Middleware System: Composable request and response middleware
Why scurl?
scurl extracts clean, readable content from web pages - perfect for LLM consumption, readability, or bandwidth savings.
Size Comparison
| Website | curl | scurl | Reduction |
|---|---|---|---|
| example.com | 513 | 167 | 67.4% |
| news.ycombinator.com | 34,082 | 10,739 | 68.5% |
| en.wikipedia.org/wiki/Curl | 110,373 | 10,044 | 90.9% |
| github.com/anthropics | 296,788 | 353 | 99.9% |
| docs.python.org | 319,554 | 12,348 | 96.1% |
Visual Comparison
curl output (Wikipedia, first 500 chars):
<!DOCTYPE html><html class="client-nojs" lang="en" dir="ltr"><head>
<meta charset="UTF-8"/><title>Curl (programming language) - Wikipedia</title>
<script>(function(){var className="client-js";var cookie=document.cookie.
match(/(?:^|; )enwikimwclientpreferences=([^;]+)/);if(cookie){cookie[1].
split('%2C').forEach(function(pref){className=className.replace(new
RegExp('(^| )'+pref.replace(/-hierarchical-hierarchical/,'')
+'($| )'),'$1teleported-hierarchical$2');});...
scurl output (same page):
# Curl (programming language) - Wikipedia
**Curl** is a reflective object-oriented programming language for interactive
web applications, whose goal is to provide a smoother transition between
content formatting and computer programming. It makes it possible to embed
complex objects in simple documents without needing to switch between
programming languages or development platforms.
The Curl implementation initially consisted of an interpreter only; a compiler
was added later...
Flags
| Flag | Description |
|---|---|
--raw |
Disable all response middleware |
--disable <slug> |
Disable a middleware by slug (can be repeated) |
--enable <slug> |
Override a middleware's block (can be repeated) |
--list-middleware |
List available middleware and their slugs |
Middleware Slugs
| Slug | Type | Description |
|---|---|---|
secret-defender |
Request | Detects and blocks requests containing secrets |
readability |
Response | Extracts clean markdown from HTML |
License
Copyright 2026 Sibylline Software
MIT
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file sibylline_scurl-0.2.0.tar.gz.
File metadata
- Download URL: sibylline_scurl-0.2.0.tar.gz
- Upload date:
- Size: 45.7 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.13.11
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
758e76f1e41ff31e6a321ea8d027474648ef60221a19380f8974feace770f85b
|
|
| MD5 |
f55cfb646ec70d4c15b3e4dac995a077
|
|
| BLAKE2b-256 |
fcf6cbf4173dfda0e03dcc7df6356c64a9f06985d09ac9c2d18597688778ee92
|
File details
Details for the file sibylline_scurl-0.2.0-py3-none-any.whl.
File metadata
- Download URL: sibylline_scurl-0.2.0-py3-none-any.whl
- Upload date:
- Size: 443.0 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.13.11
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
d92e0e013e3c750019e49f0b0d4f78ff25abeec8f67dc6f7abcbcefd652b2142
|
|
| MD5 |
66c2466fa19c1982544306d72984a4e0
|
|
| BLAKE2b-256 |
0a0acebc3dcb70a93cd2d1646906ee885d01ab3f30111ce6a7c5dc753d79a8fb
|
