Israel Supermarket Scraper: Clients to download the data published by the supermarkets.
This is a scraper for ALL the supermarket chains listed in the GOV.IL site.
שקיפות מחירים (השוואת מחירים) - https://www.gov.il/he/departments/legalInfo/cpfta_prices_regulations
🤗 Want to support my work?
Daily Automatic Testing
The test suite is scheduled to run daily, so you can see if the supermarket chains have changed something in their interface and the package will not work properly.
Status:
Notice:
- Bareket and Quik are flaky! They will not fail the testing framework, but you can still use them.
- Most chains' sites block access from outside of Israel. See Running outside Israel.
- Some chains (Victory, Mahsani Ashuk) have both a legacy source and a new API-based source. Use the
_NEW_SOURCEvariant (e.g.VICTORY_NEW_SOURCE,MAHSANI_ASHUK_NEW_SOURCE) if the primary scraper stops finding files.
Running outside Israel
A weekly job tries to download one file per chain twice, from a GitHub-hosted runner outside Israel and from the project's CI machine in Israel, and publishes the comparison. A chain that works only from Israel is geo-blocked. One that fails from Israel too is broken, so please open an issue.
See the per-chain table (or the raw JSON). To check your own machine, run python -m scripts.check_reachability probe --output mine.json and then python -m scripts.check_reachability report --outside mine.json --output-dir reachability-out from a clone.
If you are outside Israel:
- Run the scraper on a host in Israel, or route it through an Israeli proxy.
- Or skip scraping and use the daily published dataset on Kaggle: erlichsefi/israeli-supermarkets-2024.
Got a question?
You can email me at erlichsefi@gmail.com
If you think you've found a bug:
- Create issue in issue tracker to see if it's already been reported
- Please consider solving the issue by yourself and creating a pull request.
What is il_supermarket_scarper?
There are a lot of projects in GitHub trying to scrape the supermarket data, but most of them are not stable or haven't been updated for a while, it's about time there will be one codebase that does the work completely.
You only need to run the following code to get all the data currently shared by the supermarkets.
Simple disk-based scraping:
from il_supermarket_scarper import ScarpingTask
scraper = ScarpingTask()
scraper.start()
Async queue-based scraping (consume files as they are downloaded):
import asyncio
from il_supermarket_scarper import ScarpingTask, ScraperFactory
from il_supermarket_scarper.utils import _now
async def main():
scraper = ScarpingTask(
output_configuration={
"output_mode": "queue",
"queue_type": "memory",
# optional; 0 (default) is unbounded. A positive cap makes
# scrapers wait when the consumer is slower than the downloads.
# "max_queue_size": 8,
},
status_configuration={"database_type": "json", "base_path": "status_logs"},
multiprocessing=1,
enabled_scrapers=[ScraperFactory.BAREKET.name, ScraperFactory.VICTORY_NEW_SOURCE.name],
)
scraper.start(limit=1, when_date=_now())
async def consume_queue():
for name, file_output in scraper.consume().items():
async for msg in file_output.queue_handler.get_all_messages():
file_name = msg["file_name"]
file_content = msg["file_content"]
file_link = msg["file_link"]
metadata = msg["metadata"]
print(f"[{name}] {file_name} ({len(file_content)} bytes)")
await consume_queue()
scraper.join()
asyncio.run(main())
Please notice! Since new files are constantly uploaded by the supermarket to their site, you will only get the current snapshot. In order to keep getting data, you will need to run this code more than one time to get the newly uploaded files.
Quick start
il_supermarket_scarper can be installed using pip:
python3 -m pip install il-supermarket-scraper
If you want to run the latest version of the code, you can install it from the repo directly:
python3 -m pip install -U git+https://github.com/OpenIsraeliSupermarkets/israeli-supermarket-scarpers.git
# or if you don't have 'git' installed
python3 -m pip install -U https://github.com/OpenIsraeliSupermarkets/israeli-supermarket-scarpers/main
Running Docker
The docker is designed to re-run against the same configuration, in every iteration the scraper will collect the files available to download and check if the file already exists before fetching it, either by scanning the dump folder, or checking the mongo/status files.
Build yourself:
docker build -t erlichsefi/israeli-supermarket-scarpers --target prod .
or pull the existing image from docker hub:
docker pull erlichsefi/israeli-supermarket-scarpers:latest
Then running it using:
docker run -v "./dumps:/usr/src/app/dumps" \
-e ENABLED_SCRAPERS="BAREKET,YAYNO_BITAN" \ # see: il_supermarket_scarper/scrappers_factory.py
-e ENABLED_FILE_TYPES="STORE_FILE" \ # see: il_supermarket_scarper/utils/file_types.py
-e FILE_NAME_REGEX="PriceFull.*" \ # optional regex matched against file names
-e LIMIT=1 \ # number of files you would like to download (remove for unlimited)
-e TODAY="2024-10-23 14:35" \ # the date to download data from
-e OUTPUT_MODE="disk" \ # 'disk' (default) or 'queue' - where to save scraped files
-e STORAGE_PATH="./dumps" \ # (optional) custom storage path for disk mode
erlichsefi/israeli-supermarket-scarpers
For queue output mode:
docker run -e OUTPUT_MODE="queue" \
-e QUEUE_TYPE="memory" \ # 'memory' (for testing) or 'kafka'
-e SCRAPING_MAX_QUEUE_SIZE="8" \ # optional; 0 or unset is unbounded
erlichsefi/israeli-supermarket-scarpers
For Kafka queue output:
docker run -e OUTPUT_MODE="queue" \
-e QUEUE_TYPE="kafka" \
-e KAFKA_BOOTSTRAP_SERVERS="localhost:9092" \ # Kafka bootstrap servers
erlichsefi/israeli-supermarket-scarpers
Environment Variables
The following environment variables can be used to configure the scraper:
General Configuration
ENABLED_SCRAPERS: Comma-separated list of scrapers to enable. Seeil_supermarket_scarper/scrappers_factory.pyfor all available scrapers. Current options include:BAREKET,YAYNO_BITAN_AND_CARREFOUR,COFIX,CITY_MARKET_KIRYATGAT,CITY_MARKET_SHOPS,DOR_ALON,GOOD_PHARM,HAZI_HINAM,HET_COHEN,KESHET,KING_STORE,MAAYAN_2000,MAHSANI_ASHUK,MAHSANI_ASHUK_NEW_SOURCE,NETIV_HASED,MESHMAT_YOSEF_1,MESHMAT_YOSEF_2,OSHER_AD,POLIZER,RAMI_LEVY,SALACH_DABACH,SHEFA_BARCART_ASHEM,SHUFERSAL,SHUK_AHIR,STOP_MARKET,SUPER_PHARM,SUPER_YUDA,SUPER_SAPIR,FRESH_MARKET_AND_SUPER_DOSH,QUIK,TIV_TAAM,VICTORY,VICTORY_NEW_SOURCE,YELLOW,YOHANANOF,ZOL_VEBEGADOL,WOLTENABLED_FILE_TYPES: Comma-separated list of file types to download (e.g., "STORE_FILE,PRICE_FILE"). Seeil_supermarket_scarper/utils/file_types.pyfor all available types.FILE_NAME_REGEX: Regular expression matched against file names withre.search. If unset, all discovered files are eligible.LIMIT: Maximum number of files to download (optional, no limit if not specified).NUMBER_OF_PROCESSES: Number of parallel processes to use (default: 5).TODAY: Date to download data from, in format "YYYY-MM-DD HH:MM" (e.g., "2024-10-23 14:35").
Output Configuration
OUTPUT_MODE: Where to save scraped files (default: "disk")disk: Save files to local filesystemqueue: Send files to a message queue
Disk Output Mode (default)
STORAGE_PATH: Custom storage path for files (optional, uses default if not specified).
Queue Output Mode
QUEUE_TYPE: Type of queue to use (required when OUTPUT_MODE="queue")memory: In-memory queue (useful for testing)kafka: Apache Kafka message queue
SCRAPING_MAX_QUEUE_SIZE: Maximum messages held in the in-memory queue (default:0, unbounded). Same asmax_queue_sizeinoutput_configuration. When the cap is reached, scrapers wait for the consumer instead of keeping every downloaded file in memory. The end-of-stream sentinel is still queued once a slot frees.
Kafka Queue
KAFKA_BOOTSTRAP_SERVERS: Kafka bootstrap servers (default: "localhost:9092").
Contributing
Help in testing, development, documentation and other tasks is highly appreciated and useful to the project. There are tasks for contributors of all experience levels.
If you need help getting started, don't hesitate to contact me.
Development status
IL SuperMarket Scraper is beta software, as far as i see devlopment stoped until new issues will be found.
1 maintainer
Metadata
1 maintainer
Release files for il-supermarket-scraper 1.0.18
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| il_supermarket_scraper-1.0.18.tar.gz | 99.2 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| il_supermarket_scraper-1.0.18-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 225.4 kB
Release files / il_supermarket_scraper-1.0.18.tar.gz
| Download URL | il_supermarket_scraper-1.0.18.tar.gz |
|---|---|
| Size | 99.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1d3ce51d407d1a938f53d3f0555e4100490b3a643835b4e67b54db3d1e1ee71c
|
|
BLAKE2b-256 checksum How to use checksums |
bd3f406eaf89fdfd7eaf4ae0f3899abe0417387b1704d89e55ac1440adfe5155
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.9.25
|
Release files / il_supermarket_scraper-1.0.18-py3-none-any.whl
| Download URL | il_supermarket_scraper-1.0.18-py3-none-any.whl |
|---|---|
| Size | 126.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
339deccb378df321d0de6139a000842293f955dc6c6feee7b569a68148a553b1
|
|
BLAKE2b-256 checksum How to use checksums |
2d70e79710f6bca141a3b1ddb2053a69d3335e84805807fdbb12b9f7d7cb4f2a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.9.25
|
Security for il-supermarket-scraper 1.0.18
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
