CrawlKit Python SDK
Official Python SDK for CrawlKit — Web + Video Intelligence API for AI.
Installation
pip install crawlkit
Quick Start
from crawlkit import CrawlKit
# Initialize client
ck = CrawlKit(api_key="ck_free_xxx")
# Scrape a webpage
result = ck.scrape("https://vnexpress.net/thoi-su")
print(result.title)
print(result.content[:200])
# Scrape YouTube video transcript
video = ck.scrape("https://youtu.be/-Td-D-vKJDg")
print(video.content) # Full transcript
# Batch scrape multiple URLs
results = ck.batch([
"https://vnexpress.net",
"https://cafef.vn",
])
for result in results:
print(f"{result.title}: {len(result.content)} chars")
Async Support
import asyncio
from crawlkit import AsyncCrawlKit
async def main():
async with AsyncCrawlKit(api_key="ck_free_xxx") as ck:
result = await ck.scrape("https://example.com")
print(result.title)
asyncio.run(main())
Features
- 🚀 Simple & Fast — Clean API with sync and async support
- 🎥 Video Intelligence — Extract transcripts from YouTube videos
- 📄 Smart Parsing — Automatic content extraction from any webpage
- 🔄 Batch Processing — Scrape multiple URLs efficiently
- 🔗 Link Discovery — Find related links on any page
- 💪 Type Safe — Full type hints support
- 🛡️ Error Handling — Automatic retries and custom exceptions
API Reference
CrawlKit(api_key, base_url=...)
Main client class.
Methods:
scrape(url, chunk=False, chunk_size=1000, parser=None)
Scrape a single URL.
Parameters:
url(str): URL to scrapechunk(bool): Split content into chunkschunk_size(int): Size of each chunkparser(str): Specific parser to use
Returns: ScrapeResult
batch(urls, chunk=False, chunk_size=1000)
Scrape multiple URLs in one request.
Parameters:
urls(list[str]): List of URLs to scrapechunk(bool): Split content into chunkschunk_size(int): Size of each chunk
Returns: list[ScrapeResult]
discover(url, limit=20)
Discover links from a page.
Parameters:
url(str): URL to discover links fromlimit(int): Maximum number of links
Returns: list[str]
health()
Check API health status.
Returns: dict
parsers()
List available parsers.
Returns: list[ParserInfo]
usage()
Get your API usage statistics.
Returns: UsageStats
Examples
Chunked Content
result = ck.scrape(
"https://en.wikipedia.org/wiki/Python",
chunk=True,
chunk_size=500
)
for i, chunk in enumerate(result.chunks):
print(f"Chunk {i+1}: {chunk[:50]}...")
Error Handling
from crawlkit import CrawlKit, RateLimitError, AuthenticationError
try:
result = ck.scrape("https://example.com")
except RateLimitError as e:
print(f"Rate limited! Retry after {e.retry_after}s")
except AuthenticationError:
print("Invalid API key")
Context Manager
with CrawlKit(api_key="ck_free_xxx") as ck:
result = ck.scrape("https://example.com")
# Client automatically closes
Get an API Key
- Visit crawlkit.vercel.app
- Sign up for a free account
- Get your API key from the dashboard
Free tier includes:
- 100 requests/day
- Web scraping
- Video transcripts
- All parsers
License
MIT License - see LICENSE file for details.
Links
- 🌐 Website
- 📚 Documentation
- 🐛 Issues
- 💬 Support
Metadata
Release files for crawlkit 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| crawlkit-0.1.0.tar.gz | 6.9 kB | Without provenance | Details for crawlkit-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| crawlkit-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for crawlkit-0.1.0-py3-none-any.whl |
Total release size: 14.4 kB
Release files / crawlkit-0.1.0.tar.gz
| Download URL | crawlkit-0.1.0.tar.gz |
|---|---|
| Size | 6.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4ec3f6347a4b3f481f0d4d87462b8ca6a1193e065a0a64cf9d7dcab6ccc15fe1
|
|
BLAKE2b-256 checksum How to use checksums |
3ee0265087735a55ec8cfb0dcf861879819fca8a7ca69eca066154a6c8a56636
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.14
|
Release files / crawlkit-0.1.0-py3-none-any.whl
| Download URL | crawlkit-0.1.0-py3-none-any.whl |
|---|---|
| Size | 7.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
3b7b6213ecfc7df675be678c6cf9277f3293aff60894eceec4dd61a38d4db120
|
|
BLAKE2b-256 checksum How to use checksums |
e2d7d97a3535799ddfe2f997a62ef5c2bca3ba5c1a68a1c43f75dc7cabab8801
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.14
|
Security for crawlkit 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.