Skip to main content

python package that implement a scraping for israeli supermarket data

Project description

Israel Supermarket Scraper: Clients to download the data published by the supermarkets.

This is a scraper for ALL the supermarket chains listed in the GOV.IL site.

שקיפות מחירים (השוואת מחירים) - https://www.gov.il/he/departments/legalInfo/cpfta_prices_regulations

Unit & Integration Tests CodeQL Pylint Publish Docker image Upload Python Package


Scheduled Automatic Testing: The test-suite is scheduled to run every three days, so you can see if the supermarket chains has chanced something in their interface and the package will not work probably. however, the tests running here are also integration tests with the APIs which may fail.

Status: Scheduled Tests

Notice:

  • Berekt and Quik are flaky! They will not fail the testing framework, but you can still use them.
  • some of the scrapers site are blocked to be accessed from outside of israel.

Got a question?

You can email me at erlichsefi@gmail.com

If you think you've found a bug:

  • Create issue in issue tracker to see if it's already been reported
  • Please consider solving the issue by yourself and creating a pull request.

What is il_supermarket_scarper?

There are alot of projects in github trying to scrape the supermarket data, most of them are not stable or wasn't updated for a while, it's about time there will be one codebase that does the work completely.

You only need to run the following code to get all the data currently shared by the supermarkets.

from il_supermarket_scarper import MainScrapperRunner

scraper = MainScrapperRunner()
scraper.run()

Please notice! since new files are constantly uploaded by the supermarket to their site, you will only get the current snapshot. In order to keep geting data, you will need to run this code more the one time to get the newly uploaded files.

Quick start

il_supermarket_scarper can be installed using pip:

python3 -m pip install -U il-supermarket-scraper

If you want to run the latest version of the code, you can install from the repo directly:

python3 -m pip install -U git+https://github.com/erlichsefi/israeli-supermarket-scarpers.git
# or if you don't have 'git' installed
python3 -m pip install -U https://github.com/erlichsefi/israeli-supermarket-scarpers/master

Running Docker

The docker is designed to run the scaper every 6 hours, (you change the cron expression if you would like, checkout the file 'crontab'), in every itreation the scraper will collect the files avaliabe to download and check if the file alreay exists before fetching it, either by scaning the dump folder, or checking the mongo.

docker-compose up -d

or if you want to use the existing image from docker hub:

docker pull erlichsefi/israeli-supermarket-scarpers:latest

Contributing

Help in testing, development, documentation and other tasks is highly appreciated and useful to the project. There are tasks for contributors of all experience levels.

If you need help getting started, don't hesitate to contact me.

Development status

IL SuperMarket Scraper is beta software, as far as i see devlopment stoped until new issues will be found.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

il-supermarket-scraper-0.3.7.tar.gz (31.3 kB view details)

Uploaded Source

Built Distribution

il_supermarket_scraper-0.3.7-py3-none-any.whl (47.1 kB view details)

Uploaded Python 3

File details

Details for the file il-supermarket-scraper-0.3.7.tar.gz.

File metadata

File hashes

Hashes for il-supermarket-scraper-0.3.7.tar.gz
Algorithm Hash digest
SHA256 bf08cc999746c0cc535bb0e5786ad9f0a5da73d67bea124b0aba3a62e065a535
MD5 caa64cf525738228031cd8ac69d83152
BLAKE2b-256 e84af64f558ea4707cb392e6f77e2c2426c619c8782e79331e7d82be157d2933

See more details on using hashes here.

File details

Details for the file il_supermarket_scraper-0.3.7-py3-none-any.whl.

File metadata

File hashes

Hashes for il_supermarket_scraper-0.3.7-py3-none-any.whl
Algorithm Hash digest
SHA256 36762da09129c7ce8e4835856171d113a380857770b173e63f91672fed6c5e96
MD5 e39cb794dd4618d0099d5db0682a287e
BLAKE2b-256 e5d4c2d34b836e484263ad9b6e26824df956faf299704ada45f4c047913ad78f

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page