6 projects
trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML.
htmldate
Fast and robust extraction of original and updated publication dates from URLs and web pages.
py3langid
Fork of the language identification tool langid.py, featuring a modernized codebase and faster execution times.
simplemma
Fast and zero-dependency lemmatization, tokenization and sentence splitting for 54 languages.
courlan
Clean, filter and sample URLs to optimize data collection – includes spam, content type and language filters.
shoten
...
