spider-py
The spider project ported to Python.
Getting Started
pip install spider_rs
import asyncio
from spider_rs import Website
async def main():
website = Website("https://choosealicense.com")
website.crawl()
print(website.get_links())
asyncio.run(main())
View the examples to learn more.
Development
Install maturin pipx install maturin and python.
maturin develop
Benchmarks
View the benchmarks to see a breakdown between libs and platforms.
Test url: https://espn.com
libraries |
pages |
speed |
|---|---|---|
spider(rust): crawl |
150,387 |
1m |
spider(nodejs): crawl |
150,387 |
153s |
spider(python): crawl |
150,387 |
186s |
scrapy(python): crawl |
49,598 |
1h |
crawlee(nodejs): crawl |
18,779 |
30m |
The benches above were ran on a mac m1, spider on linux arm machines performs about 2-10x faster.
Issues
Please submit a Github issue for any issues found.
1 maintainer
Release files for spider-rs 0.0.57
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| spider_rs-0.0.57.tar.gz | 50.2 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| spider_rs-0.0.57-cp313-cp313-macosx_11_0_arm64.whl | CPython 3.13 | CPython 3.13 | macOS 11.0+ ARM64 | Details |
Total release size: 12.3 MB
Release files / spider_rs-0.0.57.tar.gz
| Download URL | spider_rs-0.0.57.tar.gz |
|---|---|
| Size | 50.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
34b95a194c74182f65974f89ba1a156c342f2eacedc1ab339aacd913677cba5e
|
|
BLAKE2b-256 checksum How to use checksums |
126b0da3c8484c7247f7c6d1f8131336cb26ec879be5264d5c9351aa286fb108
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.8.1
|
Release files / spider_rs-0.0.57-cp313-cp313-macosx_11_0_arm64.whl
| Download URL | spider_rs-0.0.57-cp313-cp313-macosx_11_0_arm64.whl |
|---|---|
| Size | 12.2 MB |
| Tags | CPython 3.13 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
0c67c9766badae64d82acc4e06104af80de27ff53c71345a0de1acf6b8aa63d9
|
|
BLAKE2b-256 checksum How to use checksums |
e379bd2fd2d585f33dba91893f997637fb84e2e183d43a25a98912c2d5c9f88f
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
maturin/1.8.1
|
