Ecoindex Scraper
This module provides a simple interface to get the Ecoindex of a given webpage using module ecoindex-compute
Requirements
Install
pip install ecoindex_scraper
python -m camoufox fetch
If you need to convert the screenshot to webp with the generate_screenshot method, you need to install the Pillow dependency.
pip install ecoindex_scraper[webp] # OR directly pillow
Use
Get a page analysis
You can run a page analysis by calling the function get_page_analysis():
(function) get_page_analysis: (url: AnyHttpUrl, window_size: WindowSize | None = WindowSize(width=1920, height=1080), wait_before_scroll: int | None = 1, wait_after_scroll: int | None = 1) -> Coroutine[Any, Any, Result]
Example:
import asyncio
from pprint import pprint
from ecoindex.scraper import EcoindexScraper
pprint(
asyncio.run(
EcoindexScraper(url="http://ecoindex.fr").get_page_analysis()
)
)
Result example:
Result(width=1920, height=1080, url=AnyHttpUrl('http://ecoindex.fr', ), size=549.253, nodes=52, requests=12, grade='A', score=90.0, ges=1.2, water=1.8, ecoindex_version='5.0.0', date=datetime.datetime(2022, 9, 12, 10, 54, 46, 773443), page_type=None)
Default behaviour: By default, the page analysis simulates:
- Window size of 1920x1080 pixels (can be set with parameter
window_size)- Wait for 1 second when page is loaded (can be set with parameter
wait_before_scroll)- Scroll to the bottom of the page (if it is possible)
- Wait for 1 second after having scrolled to the bottom of the page (can be set with parameter
wait_after_scroll)
Get a page analysis and generate a screenshot
It is possible to generate a screenshot of the analyzed page by adding a ScreenShot property to the EcoindexScraper object.
You have to define an id (can be a string, but it is recommended to use a unique id) and a path to the screenshot file (if the folder does not exist, it will be created).
import asyncio
from pprint import pprint
from uuid import uuid1
from ecoindex.models import ScreenShot
from ecoindex.scrap import EcoindexScraper
pprint(
asyncio.run(
EcoindexScraper(
url="http://www.ecoindex.fr/",
screenshot=ScreenShot(id=str(uuid1()), folder="./screenshots"),
)
.get_page_analysis()
)
)
Async analysis
You can also run the analysis asynchronously:
import asyncio
from concurrent.futures import ThreadPoolExecutor, as_completed
from ecoindex.scrap import EcoindexScraper
def run_page_analysis(url):
return asyncio.run(
EcoindexScraper(url=url)
.get_page_analysis()
)
with ThreadPoolExecutor(max_workers=8) as executor:
future_to_analysis = {}
url = "https://www.ecoindex.fr"
for i in range(10):
future_to_analysis[
executor.submit(
run_page_analysis,
url,
)
] = (url)
for future in as_completed(future_to_analysis):
try:
print(future.result())
except Exception as e:
print(e)
Get requests details from an analysis
You can get the details of the requests made by the page by calling the function get_all_requests() and also get the aggregation of requests by category by calling the function get_requests_by_category() or by domain with get_requests_by_domain():
import asyncio
from pprint import pprint
from ecoindex.scraper import EcoindexScraper
scraper = EcoindexScraper(url="http://www.ecoindex.fr")
result = asyncio.run(scraper.get_page_analysis())
all_requests = asyncio.run(scraper.get_all_requests())
requests_by_category = asyncio.run(scraper.get_requests_by_category())
requests_by_domain = asyncio.run(scraper.get_requests_by_domain())
pprint([request.model_dump() for request in all_requests])
# [{'category': 'html',
# 'domain': 'www.ecoindex.fr',
# 'mime_type': 'text/html; charset=iso-8859-1',
# 'size': 475.0,
# 'status': 301,
# 'url': 'http://www.ecoindex.fr/'},
# {'category': 'html',
# 'domain': 'www.ecoindex.fr',
# 'mime_type': 'text/html',
# 'size': 7772.0,
# 'status': 200,
# 'url': 'https://www.ecoindex.fr/'},
# {'category': 'css',
# 'domain': 'www.ecoindex.fr',
# 'mime_type': 'text/css',
# 'size': 9631.0,
# 'status': 200,
# 'url': 'https://www.ecoindex.fr/css/bundle.min.d38033feecefa0352173204171412aec01f58eee728df0ac5c917a396ca0bc14.css'},
# {'category': 'javascript',
# 'domain': 'www.ecoindex.fr',
# 'mime_type': 'application/javascript',
# 'size': 9823.0,
# 'status': 200,
# 'url': 'https://www.ecoindex.fr/fr/js/bundle.8781a9ae8d87b4ebaa689167fc17b7d71193cf514eb8bb40aac9bf4548e14533.js'},
# {'category': 'other',
# 'domain': 'www.ecoindex.fr',
# 'mime_type': 'x-unknown',
# 'size': 892.0,
# 'status': 200,
# 'url': 'https://www.ecoindex.fr/images/logo-neutral-it.webp'},
# {'category': 'image',
# 'domain': 'www.ecoindex.fr',
# 'mime_type': 'image/svg+xml',
# 'size': 3298.0,
# 'status': 200,
# 'url': 'https://www.ecoindex.fr/images/logo-greenit.svg'}]
pprint(requests_by_category.model_dump())
# {'css': {'total_count': 1, 'total_size': 9631.0},
# 'font': {'total_count': 0, 'total_size': 0.0},
# 'html': {'total_count': 2, 'total_size': 8247.0},
# 'image': {'total_count': 1, 'total_size': 3298.0},
# 'javascript': {'total_count': 1, 'total_size': 9823.0},
# 'other': {'total_count': 1, 'total_size': 892.0},
# 'video': {'total_count': 0, 'total_size': 0.0}}
pprint({domain: metrics.model_dump() for domain, metrics in requests_by_domain.items()})
# {'www.ecoindex.fr': {'total_count': 6, 'total_size': 31811.0}}
1 maintainer
Metadata
1 maintainer
Release files for ecoindex_scraper 3.19.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| ecoindex_scraper-3.19.0.tar.gz | 21.3 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| ecoindex_scraper-3.19.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 50.3 kB
Release files / ecoindex_scraper-3.19.0.tar.gz
| Download URL | ecoindex_scraper-3.19.0.tar.gz |
|---|---|
| Size | 21.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1cf4e43669fb2b142b7cebfc471a54580532a56bd6f1b217baa674c8cf5d86b0
|
|
BLAKE2b-256 checksum How to use checksums |
2faab5b3fb74f299bcbc4a08ff2b32668682524faa0b5a129a353c56b5d5b403
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.23 {"installer":{"name":"uv","version":"0.12.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / ecoindex_scraper-3.19.0-py3-none-any.whl
| Download URL | ecoindex_scraper-3.19.0-py3-none-any.whl |
|---|---|
| Size | 28.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
48b6c2f180089d71db3c39756bd4a6b01de80214e5382ba50df121ababedf3e6
|
|
BLAKE2b-256 checksum How to use checksums |
89917ed93a8cf068519a2f12578fad5a71a998b42a6674c33ae35c0588da7c4e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.12.23 {"installer":{"name":"uv","version":"0.12.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
