Skip to main content

Python package, scraping recipes from all over the internet

Project description

Version Downloads License GitHub Actions Unittests Coveralls GitHub Actions Linters Black formatted Github

A simple web scraping tool for recipe sites.

pip install recipe-scrapers

then:

from recipe_scrapers import scrape_me

# give the url as a string, it can be url from any site listed below
scraper = scrape_me('https://www.allrecipes.com/recipe/158968/spinach-and-feta-turkey-burgers/')

# Q: What if the recipe site I want to extract information from is not listed below?
# A: You can give it a try with the wild_mode option! If there is Schema/Recipe available it will work just fine.
scraper = scrape_me('https://www.feastingathome.com/tomato-risotto/', wild_mode=True)

scraper.title()
scraper.total_time()
scraper.yields()
scraper.ingredients()
scraper.instructions()
scraper.image()
scraper.host()
scraper.links()
scraper.nutrients()  # if available

Notes:

  • Starting from v13.0.0 the packaged stopped suppressing scraper exceptions by default. If you want the previous behaviour

import os
from recipe_scrapers import scrape_me

os.environ["RECIPE_SCRAPERS_SETTINGS"] = "recipe_scrapers.settings.v12_settings"

scraper = scrape_me(...)  # etc.
  • scraper.links() returns a list of dictionaries containing all of the <a> tag attributes. The attribute names are the dictionary keys.

Scrapers available for:

Contribute

Part of the reason I want this open sourced is because if a site makes a design change, the scraper for it should be modified.

If you spot a design change (or something else) that makes the scraper unable to work for a given site - please fire an issue asap.

If you are programmer PRs with fixes are warmly welcomed and acknowledged with a virtual beer.

If you want a scraper for a new site added

  • Open an Issue providing us the site name, as well as a recipe link from it.

  • You are a developer and want to code the scraper on your own:

    • If Schema is available on the site - you can do this

    • Otherwise, scrape the HTML - like this

    • Generating a new scraper class:

      python generate.py <ClassName> <URL>
      • ClassName: The name of the new scraper class.

      • URL: The URL of an example recipe from the target site. The content will be stored in test_data to be used with the test class.

For Devs / Contribute

Assuming you have python3 installed, navigate to the directory where you want this project to live in and drop these lines

git clone git@github.com:hhursev/recipe-scrapers.git &&
cd recipe-scrapers &&
python3 -m venv .venv &&
source .venv/bin/activate &&
pip install -r requirements-dev.txt &&
pre-commit install &&
python -m coverage run -m unittest &&
python -m coverage report

In case you want to run a single unittest for a newly developed scraper

python -m coverage run -m unittest tests.test_myscraper

FAQ

  • How do I know if a website has a Recipe Schema? Run in python shell:

from recipe_scrapers import scrape_me
scraper = scrape_me('<url of a recipe from the site>', wild_mode=True)
# if no error is raised - there's schema available:
scraper.title()
scraper.instructions()  # etc.

Special thanks to:

All the contributors that helped improving the package. You are awesome!

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

recipe_scrapers-13.4.0.tar.gz (50.7 kB view details)

Uploaded Source

Built Distribution

recipe_scrapers-13.4.0-py3-none-any.whl (320.0 kB view details)

Uploaded Python 3

File details

Details for the file recipe_scrapers-13.4.0.tar.gz.

File metadata

  • Download URL: recipe_scrapers-13.4.0.tar.gz
  • Upload date:
  • Size: 50.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.4 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.2 CPython/3.9.6

File hashes

Hashes for recipe_scrapers-13.4.0.tar.gz
Algorithm Hash digest
SHA256 40296e0e77cb45018ada196dc1d3d58a8fc37e52361ba11b0b0ab286940406c0
MD5 0fcffc3098e6caff285dba4ecadb6b63
BLAKE2b-256 ef47e3c5c18d6cf543b04288cac79ce2e9bc1dfe58cb217a878de93662fb2599

See more details on using hashes here.

File details

Details for the file recipe_scrapers-13.4.0-py3-none-any.whl.

File metadata

  • Download URL: recipe_scrapers-13.4.0-py3-none-any.whl
  • Upload date:
  • Size: 320.0 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.6.4 pkginfo/1.7.1 requests/2.26.0 requests-toolbelt/0.9.1 tqdm/4.62.2 CPython/3.9.6

File hashes

Hashes for recipe_scrapers-13.4.0-py3-none-any.whl
Algorithm Hash digest
SHA256 f38a0c6c0e6394ce3062e715ac43a6d1b40c0d7930d0ec92849a154f7b0501ad
MD5 5abb17bcfc36ae3c03337be0747fec6b
BLAKE2b-256 8e1c2b26ec828fac9a9c2973a28eec7726aafe53ccbdf7a5f811487b419f51c4

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page