Skip to main content

An Easy-to-use and Fast Python Spider Framework

Project description

Distributed - Asynchronous - Easy-to-use

An Easy-to-use and Fast Python Spider Framework


Overview

  • AirSpider is a high-performance asynchronous crawler framework for developers
  • Based on Redis: task distribution, task deduplication, and distributed ☁️

Requirements

  • Python 3.6
  • Works on Linux, Windows, macOS

Features

  • Quick to Start
  • Low Coupling
  • High Cohesion
  • Easy Expansion
  • Orderly Workflow

Installation

# For Linux && MacOS
pip3 install airspider


# For Windows
pip3 install airspider

Documents

Topics

TODO

  • Complete Plugins of Redis
  • Complete Distributed Architecture

Contributing

AirSpider🕷️ is still under Developing

Feel free to open issues💬 and pull requests

  • Report or Fix bugs
  • Build Powerful plugins
  • Make documentation Better
  • Add Examples of Spiders

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

AirSpider-2.0.4.tar.gz (16.0 kB view details)

Uploaded Source

Built Distribution

AirSpider-2.0.4-py2.py3-none-any.whl (18.8 kB view details)

Uploaded Python 2 Python 3

File details

Details for the file AirSpider-2.0.4.tar.gz.

File metadata

  • Download URL: AirSpider-2.0.4.tar.gz
  • Upload date:
  • Size: 16.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.6.0 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.7.3

File hashes

Hashes for AirSpider-2.0.4.tar.gz
Algorithm Hash digest
SHA256 fbce74e8d154e1136260db7bd5f674feddc1e50327d9e5888ec570feaf76e8d7
MD5 a324879ef98ea2967b8e1e122f9a7cc8
BLAKE2b-256 6da5e5cbbde0cc14fc5a0c8afd13541fa796d0d76b657295b5615afa75e9eda6

See more details on using hashes here.

File details

Details for the file AirSpider-2.0.4-py2.py3-none-any.whl.

File metadata

  • Download URL: AirSpider-2.0.4-py2.py3-none-any.whl
  • Upload date:
  • Size: 18.8 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.2.0 pkginfo/1.5.0.1 requests/2.24.0 setuptools/49.6.0 requests-toolbelt/0.9.1 tqdm/4.48.2 CPython/3.7.3

File hashes

Hashes for AirSpider-2.0.4-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 f0d31596f3582e6103284b48fb35daec75f691037c66f2b167bbadfc5935cf88
MD5 733c9fc9262cf1394911a1446632ef9d
BLAKE2b-256 d844c6cfccfaf0d21d662a2fc356c4394573703379ef55da31d3225e1ba65218

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page