6 projects
trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML.
htmldate
Fast and robust extraction of original and updated publication dates from URLs and web pages.
courlan
Clean, filter and sample URLs to optimize data collection – includes spam, content type and language filters.
simplemma
A lightweight toolkit for multilingual lemmatization and language detection.
py3langid
Fork of the language identification tool langid.py, featuring a modernized codebase and faster execution times.
shoten
...
