new_frontera
Overview
new_frontera is a web crawling framework consisting of crawl frontier, and distribution/scaling primitives, allowing to build a large scale online web crawler.
new_frontera takes care of the logic and policies to follow during the crawl. It stores and prioritizes links extracted by the crawler to decide which pages to visit next, and capable of doing it in distributed manner.
Main features
- Online operation: small requests batches, with parsing done right after fetch.
- Pluggable backend architecture: low-level backend access logic is separated from crawling strategy.
- Two run modes: single process and distributed.
- Built-in SqlAlchemy, Redis and HBase backends.
- Built-in Apache Kafka and ZeroMQ message buses.
- Built-in crawling strategies: breadth-first, depth-first, Discovery (with support of robots.txt and sitemaps).
- Battle tested: our biggest deployment is 60 spiders/strategy workers delivering 50-60M of documents daily for 45 days, without downtime,
- Transparent data flow, allowing to integrate custom components easily using Kafka.
- Message bus abstraction, providing a way to implement your own transport (ZeroMQ and Kafka are available out of the box).
- Optional use of Scrapy for fetching and parsing.
- 3-clause BSD license, allowing to use in any commercial product.
- Python 3 support.
Installation
Development version:
$ pip install git+https://github.com/ZeroCool940711/new_frontera.git
or from PyPi:
$ pip install new-frontera
Documentation
Community
Join our Google group at https://groups.google.com/a/scrapinghub.com/forum/#!forum/frontera or check GitHub issues and pull requests.
1 maintainer
Metadata
1 maintainer
Release files for new-frontera 0.9.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| new_frontera-0.9.0.tar.gz | 128.0 kB | Without provenance | Details for new_frontera-0.9.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| new_frontera-0.9.0-py3-none-any.whl | Python 3 | none | any | Details for new_frontera-0.9.0-py3-none-any.whl |
Total release size: 253.2 kB
Release files / new_frontera-0.9.0.tar.gz
| Download URL | new_frontera-0.9.0.tar.gz |
|---|---|
| Size | 128.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
36fbbfa932c2799463abd2f51b9296410c08f879044000c78d65c9efaeda731e
|
|
BLAKE2b-256 checksum How to use checksums |
66575731e4d6fe79f265ea113c457f4c75c53b717d863d26de5b48a5cb1391f3
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.18
|
Release files / new_frontera-0.9.0-py3-none-any.whl
| Download URL | new_frontera-0.9.0-py3-none-any.whl |
|---|---|
| Size | 125.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6a6c1dd1196cf0fab235ecbc058a91a2c344231c2500f5c98d37713df01cc4a2
|
|
BLAKE2b-256 checksum How to use checksums |
ab72aed55c3d88901f8bb7b83ca83daf73ecc9336b1a0561ec1e89e33eeaf9ff
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.9.18
|
Security for new-frontera 0.9.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
