spider-rs 0.0.1

The fastest web crawler written in Rust ported to nodejs.

spider-py

The spider project ported to Python (Incomplete Port WIP).

Getting Started

1. pip install spider_rs

import asyncio

from spider_rs import crawl

async def main():
    website = await crawl("https://choosealicense.com")
    print(website.links)
    # print(website.pages)

asyncio.run(main())

Development

Install maturin pipx install maturin and python.

1. maturin develop

Todo

1. Add thread safe callback handling crawl/scrape.
2. Add callback Cron.
3. Add subscription callback.

Once these items are done the base of the module should be complete. Most of the code comes from the initial port to Node.js that was done.