6 projects
trafilatura
Python package and command-line tool designed to gather text on the Web, includes all necessary discovery and text processing components to perform web crawling, downloads, scraping, and extraction of main texts, metadata and comments.
py3langid
Fork of the language identification tool langid.py, featuring a modernized codebase and faster execution times.
courlan
Clean, filter and sample URLs to optimize data collection – includes spam, content type and language filters.
simplemma
A simple multilingual lemmatizer for Python.
htmldate
Fast and robust extraction of original and updated publication dates from URLs and web pages.
shoten
...