a scraper to mirror edi-energy.de
Project description
edi-energy.de scraper
The Python package edi_energy_scraper
provides easy to use methods to mirror the website edi-energy.de.
Rationale / Why?
If you'd like to be informed about new regulations or data formats being published on edi-energy.de you can either
- visit the site every day and hope that you see the changes if this is your favourite hobby,
- or automate the task.
This repository helps you with the latter. It allows you to create an up-to-date copy of edi-energy.de on your local
computer. Other than if you mirrored the files using wget
or curl
, you'll get a clean and intuitive directory
structure.
From there you can e.g. commit the files into a VCS (like e.g. our edi_energy_mirror), scrape the PDF/Word files for later use...
We're all hoping for the day of true digitization on which this repository will become obsolete.
How to use the Package (as a user)
Install via pip:
pip install edi_energy_scraper
Create a directory in which you'd like to save the mirrored data:
mkdir edi_energy_de
Then import it and start the download:
import asyncio
from edi_energy_scraper import EdiEnergyScraper
# add the following lines to enable debug logging to stdout (CLI)
# import logging
# import sys
# logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
async def mirror():
scraper = EdiEnergyScraper(path_to_mirror_directory="edi_energy_de")
await scraper.mirror()
if __name__ == "__main__":
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
asyncio.run(mirror())
This creates a directory structure:
-|-your_script_cwd.py
|-edi_energy_de
|- past (contains archived files)
|- ahb.pdf
|- ahb.docx
|- ...
|- current (contains files valid as of today)
|- mig.pdf
|- mig.docx
|- ...
|- future (contains files valid in the future)
|- allgemeine_festlegungen.pdf
|- schema.xsd
|- ...
How to use this Repository on Your Machine (for development)
Please follow the instructions in our Python Template Repository . And for further information, see the Tox Repository.
Contribute
You are very welcome to contribute to this template repository by opening a pull request against the main branch.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for edi_energy_scraper-0.7.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 835216171bffa053ef77d2164cf65b862c5a8d89ed1dded43407201a558cffd2 |
|
MD5 | 6e66991237b712a252462af00a66dd08 |
|
BLAKE2b-256 | 1fae95a88ca42f524b5416764b88db79507a6226a6a4f7967683b287e188786e |