scrapeMM: Multimodal Web Retrieval
Simple web scraper to asynchronously retrieve webpages and access social media contents, fetching text along with media, i.e., images and videos.
This library aims to help developers and researchers to easily access multimodal data from the web and use it for LLM processing.
Setup
- If you want to download videos: Then, the installation of ffmpeg is highly recommended.
In Conda, you can install it with
conda install -c conda-forge ffmpeg. - If you want to scrape Perma.cc archive records or Facebook photos, you'll need to install playwright with
pip install playwrightand runningplaywright install.
Configure
To set the API secrets, run
from scrapemm import configure_secrets
configure_secrets()
To set the Firecrawl URL, run
from scrapemm import update_config
update_config(firecrawl_url="your_url")
Usage
from scrapemm import retrieve
import asyncio
if __name__ == "__main__":
url = "https://www.snopes.com/fact-check/gauze-originate-from-gaza/"
result = asyncio.run(retrieve(url))
if result.errors:
print(result.errors)
else:
print(result.content)
scrapeMM will ask you for the API secrets needed for the integrations. You may skip them if you don't need them.
You will also be prompted to choose a password that is used to secure the secrets in an encrypted file.
How it works
Input: Output:
URL (string) --> retrieve() --> MultimodalSequence
The MultimodalSequence is a sequence of Markdown-formatted text and media provided by the ezMM library.
Web scraping is done with Firecrawl and Decodo.
Supported Platforms
Social Media
- ✅ X/Twitter
- ✅ Telegram
- ✅ Bluesky
- ✅ TikTok
- ✅ YouTube
- (✅️) Instagram: works for most content
- ✅️ Facebook
- ❌ Threads: TBD
- ❌ Reddit: TBD
Archiving Services
- ✅ Perma.cc
- ✅ Archive.today: Rarely ending up in TimeoutErrors
- ✅ MediaVault (mvau.lt)
- ✅ Internet Archive (web.archive.org)
- ✅ AwesomeScreenshot.com
- ✅ Ghostarchive (ghostarchive.org)
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file scrapemm-0.7.4.tar.gz.
File metadata
- Download URL: scrapemm-0.7.4.tar.gz
- Upload date:
- Size: 64.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.1.0 CPython/3.10.11
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
3fa95437a54179e0860d1faf6a919fbba4e74b815f13c1482d0532963c423588
|
|
| MD5 |
27f21ba3c8c87fb1d3513dfe3233348e
|
|
| BLAKE2b-256 |
453e6d427dacb26fb6bdfb741d158a8a208520eb55649cbef8d13c92fb6bb0ac
|
File details
Details for the file scrapemm-0.7.4-py3-none-any.whl.
File metadata
- Download URL: scrapemm-0.7.4-py3-none-any.whl
- Upload date:
- Size: 77.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/6.1.0 CPython/3.10.11
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
e2a9c7d274dbf8f92ef4dcc7171d510f1b3d47a37fb1828609c245d5b5a566be
|
|
| MD5 |
70348e213af62556ec7740ea529e2e26
|
|
| BLAKE2b-256 |
bec4f77ca7ba9ae6eca337bef5ff31638d7b8d77e81f30ac25141e7d8b372090
|
