A simple web scraping tool for recipe sites.
pip install recipe-scrapersthen:
from recipe_scrapers import scrape_me
scraper = scrape_me('https://www.allrecipes.com/recipe/158968/spinach-and-feta-turkey-burgers/')
# Q: What if the recipe site I want to extract information from is not listed below?
# A: You can give it a try with the wild_mode option! If there is Schema/Recipe available it will work just fine.
scraper = scrape_me('https://www.feastingathome.com/tomato-risotto/', wild_mode=True)
scraper.host()
scraper.title()
scraper.total_time()
scraper.image()
scraper.ingredients()
scraper.ingredient_groups()
scraper.instructions()
scraper.instructions_list()
scraper.yields()
scraper.to_json()
scraper.links()
scraper.nutrients() # if availableYou also have an option to scrape html-like content
import requests
from recipe_scrapers import scrape_html
url = "https://www.allrecipes.com/recipe/158968/spinach-and-feta-turkey-burgers/"
html = requests.get(url).content
scraper = scrape_html(html=html, org_url=url)
scraper.title()
scraper.total_time()
# etc...Notes:
scraper.links() returns a list of dictionaries containing all of the <a> tag attributes. The attribute names are the dictionary keys.
Scrapers available for:
(*) offline saved files only
Contribute
If you spot a design change (or something else) that makes the scraper unable to work for a given site - please fire an issue asap.
If you are programmer PRs with fixes are warmly welcomed and acknowledged with a virtual beer.
If you want a scraper for a new site added
Open an Issue providing us the site name, as well as a recipe link from it.
You are a developer and want to code the scraper on your own:
If Schema is available on the site - you can go like this.
Otherwise, scrape the HTML - like this
Generating a new scraper class:
python generate.py <ClassName> <URL>ClassName: The name of the new scraper class.
URL: The URL of an example recipe from the target site. The content will be stored in test_data to be used with the test class.
For Devs / Contribute
Assuming you have >=python3.7 installed, navigate to the directory where you want this project to live in and drop these lines
git clone git@github.com:hhursev/recipe-scrapers.git &&
cd recipe-scrapers &&
python -m venv .venv &&
source .venv/bin/activate &&
python -m pip install --upgrade pip &&
pip install -r requirements-dev.txt &&
pip install pre-commit &&
pre-commit install &&
python -m unittestIn case you want to run a single unittest for a newly developed scraper
python -m unittest tests.test_myscraperFAQ
How do I know if a website has a Recipe Schema? Run in python shell:
from recipe_scrapers import scrape_me
scraper = scrape_me('<url of a recipe from the site>', wild_mode=True)
# if no error is raised - there's schema available:
scraper.title()
scraper.instructions() # etc.Netiquette
If you’re using this library to collect large numbers of recipes from the web, please use the software responsibly and try to avoid creating high volumes of network traffic.
Python’s standard library provides a robots.txt parser that may be helpful to automatically follow common instructions specified by websites for web crawlers.
Another parser option – particularly if you find that many web requests from urllib.robotparser are blocked – is the robotexclusionrulesparser library.
Special thanks to:
All the contributors that helped improving the package. You are awesome!
Extra:
1 maintainer
Metadata
1 maintainer
Release files for recipe-scrapers 14.41.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| recipe_scrapers-14.41.0.tar.gz | 312.1 kB | Without provenance | Details for recipe_scrapers-14.41.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| recipe_scrapers-14.41.0-py3-none-any.whl | Python 3 | none | any | Details for recipe_scrapers-14.41.0-py3-none-any.whl |
Total release size: 484.3 kB
Release files / recipe_scrapers-14.41.0.tar.gz
| Download URL | recipe_scrapers-14.41.0.tar.gz |
|---|---|
| Size | 312.1 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
86653d76585a116da14e934e8d296d1221c7c290c798d7c30551a8cfdac39ade
|
|
BLAKE2b-256 checksum How to use checksums |
54fbe10afb84961ba1ea08c724a02d2ac5efc36fa85b6a02e1807bcc9769dd04
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.4
|
Release files / recipe_scrapers-14.41.0-py3-none-any.whl
| Download URL | recipe_scrapers-14.41.0-py3-none-any.whl |
|---|---|
| Size | 172.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
97926707edaf7c6ce06dfc8bb6b769dc88502a7a8c77e5bcf7a3c1faaaca1e9a
|
|
BLAKE2b-256 checksum How to use checksums |
8d2e0efe660cc5596ea6d79d5dc9955b51261c449604411f16f99bfa30955484
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.11.4
|
Security for recipe-scrapers 14.41.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
