This release is a pre-release and may not be stable for production use.
Python Web Robot Builder
The main idea of py-robot is to simplify the code, and improve the performance of web crawlers.
Install
pip install ciag-robot
Intro
Bellow we have a simple example of crawler that needs to get a page, and for each specific item get another page. Because it was written without the use of async requests, it will make a request and make the another one only when the previous has finished.
# examples/iot_eetimes.py
import requests
import json
from lxml import html
from pyquery.pyquery import PyQuery as pq
page = requests.get('https://iot.eetimes.com/')
dom = pq(html.fromstring(page.content.decode()))
result = []
for link in dom.find('.theiaStickySidebar ul li'):
news = {
'category': pq(link).find('span').text(),
'url': pq(link).find('a[href]').attr('href'),
}
news_page = requests.get(news['url'])
dom = pq(news_page.content.decode())
news['body'] = dom.find('p').text()
news['title'] = dom.find('h1.post-title').text()
result.append(news)
print(json.dumps(result, indent=4))
We can rewrite that using py-robot, and it will look like that:
# examples/iot_eetimes2.py
import json
from robot import Robot
from robot.collector.shortcut import *
import logging
logging.basicConfig(level=logging.DEBUG)
collector = pipe(
const('https://iot.eetimes.com/'),
get(),
css('.theiaStickySidebar ul li'),
foreach(dict(
pipe(
css('a[href]'), attr('href'), any(),
get(),
dict(
body=pipe(css('p'), as_text()),
title=pipe(css('h1.post-title'), as_text()),
)
),
category=pipe(css('span'), as_text()),
url=pipe(css('a[href]'), attr('href'), any(), url())
))
)
with Robot() as robot:
result = robot.sync_run(collector)
print(json.dumps(result, indent=4))
Now all the requests will be async, so it will start all the requests for each item at the same time, and it will improve the performance of the crawler.
Metadata
Release files for ciag-robot 0.4.dev1697464011
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| ciag-robot-0.4.dev1697464011.tar.gz | 15.0 kB | Without provenance | Details for ciag-robot-0.4.dev1697464011.tar.gz |
Release files / ciag-robot-0.4.dev1697464011.tar.gz
| Download URL | ciag-robot-0.4.dev1697464011.tar.gz |
|---|---|
| Size | 15.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
5e9aaea062e9b5bee73fea2cb5f1966f582516ac9ab20c3b6466dc116e1234ba
|
|
BLAKE2b-256 checksum How to use checksums |
bc488dcd6bf46220b0844e568ae6bc0fd654326bdd1d163b1df988fa2281028c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.2
|
Security for ciag-robot 0.4.dev1697464011
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.