Skip to main content

Extracts cooking recipe from HTML structured data in the format.

Project description


PyPI Build Status Downloads

Scrapes recipes from HTML (Microdata/JSON-LD) into Python dictionaries.


pip install scrape-schema-recipe


Python version 3.6+

This library relies heavily upon extruct.

Other requirements:

  • isodate (>=0.5.1)
  • requests

Online Example

>>> import scrape_schema_recipe

>>> url = ''
>>> recipe_list = scrape_schema_recipe.scrape_url(url, python_objects=True)
>>> len(recipe_list)
>>> recipe = recipe_list[0]

# Name of the recipe
>>> recipe['name']
'Honey Mustard Dressing'

# List of the Ingredients
>>> recipe['recipeIngredient']
['5 tablespoons medium body honey (sourwood is nice)',
 '3 tablespoons smooth Dijon mustard',
 '2 tablespoons rice wine vinegar']

# List of the Instructions
>>> recipe['recipeInstructions']
['Combine all ingredients in a bowl and whisk until smooth. Serve as a dressing or a dip.']

# Author
>>> recipe['author']
[{'@type': 'Person',
  'name': 'Alton Brown',
  'url': ''}]

'@type': 'Person' is a object

# Preparation Time
>>> recipe['prepTime']
datetime.timedelta(0, 300)

# The library pendulum can give you something a little easier to read.
>>> import pendulum

# for pendulum version 1.0
>>> pendulum.Interval.instanceof(recipe['prepTime'])
<Interval [5 minutes]>

# for version 2.0 of pendulum
>>> pendulum.Duration(seconds=recipe['prepTime'].total_seconds())
<Duration [5 minutes]>

If python_objects is set to False, this would return the string ISO8611 representation of the duration, 'PT5M'

pendulum's library website.

# Publication date
>>> recipe['datePublished']
datetime.datetime(2016, 11, 13, 21, 5, 50, 518000, tzinfo=<FixedOffset '-05:00'>)

>>> str(recipe['datePublished'])
'2016-11-13 21:05:50.518000-05:00'

# Identifying this is data (in LD-JSON format)
 >>> recipe['@context'], recipe['@type']
('', 'Recipe')

# Content's URL
>>> recipe['url']

# all the keys in this dictionary
>>> recipe.keys()
dict_keys(['recipeYield', 'totalTime', 'dateModified', 'url', '@context', 'name', 'publisher', 'prepTime', 'datePublished', 'recipeIngredient', '@type', 'recipeInstructions', 'author', 'mainEntityOfPage', 'aggregateRating', 'recipeCategory', 'image', 'headline', 'review'])

Example from a File (alternative representations)

Also works with locally saved HTML example file.

>>> filelocation = 'test_data/google-recipe-example.html'
>>> recipe_list = scrape_schema_recipe.scrape(filelocation, python_objects=True)
>>> recipe = recipe_list[0]

>>> recipe['name']
'Party Coffee Cake'

>>> repcipe['datePublished'], 3, 10)

# Recipe Instructions using the HowToStep
>>> recipe['recipeInstructions']
[{'@type': 'HowToStep',
  'text': 'Preheat the oven to 350 degrees F. Grease and flour a 9x9 inch pan.'},
 {'@type': 'HowToStep',
  'text': 'In a large bowl, combine flour, sugar, baking powder, and salt.'},
 {'@type': 'HowToStep', 'text': 'Mix in the butter, eggs, and milk.'},
 {'@type': 'HowToStep', 'text': 'Spread into the prepared pan.'},
 {'@type': 'HowToStep', 'text': 'Bake for 30 to 35 minutes, or until firm.'},
 {'@type': 'HowToStep', 'text': 'Allow to cool.'}]

What Happens When Things Go Wrong

If there aren't any formatted recipes on the site.

>>> url = ''
>>> recipe_list = scrape_schema_recipe.scrape(url, python_objects=True)

>>> len(recipe_list)

Some websites will cause an HTTPError.

You may get around a 403 - Forbidden Errror by putting in an alternative user-agent via the variable user_agent_str.


  • load() - load HTML structured data from a file or file-like object
  • loads() - loads HTML structured data from a string
  • scrape_url() - scrape a URL for HTML structured data
  • scrape() - load HTML structured data from a file, file-like object, string, or URL
    location : string or file-like object
        A url, filename, or text_string of HTML, or a file-like object.

    python_objects : bool, list, or tuple  (optional)
        when True it translates certain data types into python objects
          dates into, datetimes into datetime.datetimes,
          durations as dateime.timedelta.
        when set to a list or tuple only converts types specified to
          python objects:
            * when set to either [] or [datetime.datetimes] either will
              convert dates.
            * when set to [datetime.timedelta] durations will be converted
        when False no conversion is performed
        (defaults to False)

    nonstandard_attrs : bool, optional
        when True it adds nonstandard (for attributes to the
        resulting dictionaries, that are outside the specification such as:
            '_format' is either 'json-ld' or 'microdata' (how was encoded into HTML)
            '_source_url' is the source url, when 'url' has already been defined as another value
        (defaults to False)

    migrate_old_schema : bool, optional
        when True it migrates the schema from older version to current version
        (defaults to True)

    user_agent_str : string, optional  ***only for scrape_url() and scrape()***
        overide the user_agent_string with this value.
        (defaults to None)

        a list of dictionaries in the style of JSON-LD
        no results - an empty list will be returned

These are also available with help() in the python console.

Example function

The example_output() function gives quick access to data for prototyping and debugging. It accepts the same parameters as load(), but the first parameter, name, is different.

>>> from scrape_schema_recipe import example_names, example_output

>>> example_names
('irish-coffee', 'google', 'taco-salad', 'tart', 'tea-cake', 'truffles')

>>> recipes = example_output('truffles')
>>> recipes[0]['name']
'Rum & Tonka Bean Dark Chocolate Truffles'


License: Apache 2.0 see LICENSE

Test data attribution and licensing:


The unit testing must be run from a copy of the repository folder. Unit testing can be run by:

schema-recipe-scraper$ python3

mypy is used for static type checking

from the project directory:

 schema-recipe-scraper$ mypy schema_recipe_scraper/

If you run mypy from another directory the --ignore-missing-imports flag will need to be added, thus $ mypy --ignore-missing-imports

--ignore-missing-imports flag is used because most libraries don't have static typing information contained in their own code or typeshed.

Reference Documentation

Here are some references for how should be structured:

Other Similar Python Libraries

  • recipe_scrapers - library scrapes recipes by using extruct to scrape the format or HTML tags with BeautifulSoup. The library has drivers that support many different websites that further parse the information. This is a solid alternative to schema-recipe-scraper that is focused on a different kind of simplicity.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

scrape-schema-recipe-0.2.0.tar.gz (15.4 kB view hashes)

Uploaded source

Built Distribution

scrape_schema_recipe-0.2.0-py2.py3-none-any.whl (277.7 kB view hashes)

Uploaded py2 py3

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page