Ruia logo

Ruia

🕸️ Async Python 3.6+ web scraping micro-framework based on asyncio.

⚡ Write less, run faster.

travis codecov PyPI - Python Version PyPI Downloads gitter

Overview

Ruia is an async web scraping micro-framework, written with asyncio and aiohttp, aims to make crawling url as convenient as possible.

Write less, run faster:

Features

  • Easy: Declarative programming
  • Fast: Powered by asyncio
  • Extensible: Middlewares and plugins
  • Powerful: JavaScript support

Installation

# For Linux & Mac
pip install -U ruia[uvloop]

# For Windows
pip install -U ruia

# New features
pip install git+https://github.com/howie6879/ruia

Tutorials

  1. Overview
  2. Installation
  3. Define Data Items
  4. Spider Control
  5. Request & Response
  6. Customize Middleware
  7. Write a Plugins

TODO

  • Cache for debug, to decreasing request limitation, ruia-cache
  • Provide an easy way to debug the script, ruia-shell
  • Distributed crawling/scraping

Contribution

Ruia is still under developing, feel free to open issues and pull requests:

  • Report or fix bugs
  • Require or publish plugins
  • Write or fix documentation
  • Add test cases

!!!Notice: We use black to format the code.

Thanks

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for howie from gravatar.com howie

Credits

Author: Howie Hu

License

MIT License (MIT)

Requires

Python >=3.6

Classifiers

Report project as malware

Release files for ruia 0.8.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ruia 0.8.5
File Size Uploaded
ruia-0.8.5.tar.gz 25.4 kB Details

Built distribution (wheel)

Table of built distributions (wheels) for ruia 0.8.5
File Interpreter ABI Platform
ruia-0.8.5-py2.py3-none-any.whl 30.3 kB Python 3, Python 2 none any Details

Total release size: 55.8 kB

Release files / ruia-0.8.5.tar.gz

Download URL ruia-0.8.5.tar.gz
Size 25.4 kB
Tags Source
SHA-256 checksum
How to use checksums 		4c94e40e48ba6829dfcc45852d6a99a1187f1c3309d8b6858c080b7844b14a72
BLAKE2b-256 checksum
How to use checksums 		de1bb3f516402d1e17f01487ae0dc19daa9c9f968beb21eb593b9903623bc5d3
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/4.0.1 CPython/3.9.7

Release files / ruia-0.8.5-py2.py3-none-any.whl

Download URL ruia-0.8.5-py2.py3-none-any.whl
Size 30.3 kB
Tags Python 2 Python 3
SHA-256 checksum
How to use checksums 		7da42157fd2a1437a4cd8dd0be1bb24a2aae1e656bdfc132a8cf1b1b7e779b46
BLAKE2b-256 checksum
How to use checksums 		12f516d254cf0a6cd1b088e56992b8a2e6f73e19f3b0b8d5024720ecf6789d4b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/4.0.1 CPython/3.9.7

Release history Release notifications | RSS feed

This release

0.8.5 This release

2 release files

0.8.4

1 release file

0.8.3

1 release file

0.8.2

2 release files

0.8.0

1 release file

0.7.1

1 release file

0.7.0

1 release file

0.6.9

1 release file

0.6.8

1 release file

0.6.7

2 release files

0.6.6

2 release files

0.6.5

2 release files

0.6.4

2 release files

0.6.3

2 release files

0.6.2

2 release files

0.6.1

2 release files

0.6.0

2 release files

0.5.9

2 release files

0.5.8

2 release files

0.5.7

2 release files

0.5.6

2 release files

0.5.5

2 release files

0.5.4

2 release files

0.5.3

2 release files

0.5.2

2 release files

0.5.1

2 release files

0.5.0

2 release files

0.4.9

2 release files

0.4.8

2 release files

0.4.7

2 release files

0.4.6

2 release files

0.4.5

2 release files

0.4.4

2 release files

0.4.3

2 release files

0.4.2

2 release files

0.4.1

2 release files

0.4.0

2 release files

0.3.3

2 release files

0.3.2

2 release files

0.3.1

2 release files

0.3.0

2 release files

0.2.1

2 release files

0.2.0

2 release files

0.1.3

2 release files

0.1.2

2 release files

0.1.1

2 release files

0.1.0

2 release files

0.0.9

2 release files

0.0.8

2 release files

0.0.7

2 release files

0.0.5

2 release files

0.0.4

2 release files

0.0.3

2 release files

0.0.2

2 release files

0.0.1

2 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page