Extraction-based Turkish news summarizer.
Project description
SadedeGel: An extraction based Turkish news summarizer
SadedeGel is a library for unsupervised extraction-based news summarization using several old and new NLP techniques.
Development of the library takes place as a part of Açık Kaynak Hackathon Programı 2020
💫 Version 0.15 out now! Check out the release notes here.
📖 Documentation
Documentation | |
---|---|
Contribute | How to contribute to the sadedeGel project and code base. |
💬 Where to ask questions
The SadedeGel project is initialized by @globalmaksimum AI team members @dafajon, @askarbozcan, @mccakir and @husnusensoy.
Other community maintainers
- @doruktiktiklar contributes TFIDF Summarizer
Type | Platforms |
---|---|
🚨 Bug Reports | GitHub Issue Tracker |
🎁 Feature Requests | GitHub Issue Tracker |
Questions | Slack Workspace |
Features
-
Several news datasets
- Basic corpus
- Raw corpus (
sadedegel.dataset.load_raw_corpus
) - Sentences tokenized corpus (
sadedegel.dataset.load_sentences_corpus
) - Human annotated summary corpus (
sadedegel.dataset.load_annotated_corpus
)
- Raw corpus (
- Extended corpus
- Raw corpus (
sadedegel.dataset.extended.load_extended_raw_corpus
) - Sentences tokenized corpus (
sadedegel.dataset.extended.load_extended_sents_corpus
)
- Raw corpus (
- Basic corpus
-
ML based sentence boundary detector (SBD) trained for Turkish language (
sadedegel.dataset
) -
Various baseline summarizers
- Position Summarizer
- First Important Summarizer
- Last Important Summarizer
- Length Summarizer
- Band Summarizer
- Random Summarizer
- Position Summarizer
-
Various unsupervised/supervised summarizers
- ROUGE1 Summarizer
- Cluster Summarizer
- Supervised Summarizer
-
Various Word Tokenizers
- BERT Tokenizer - Trained tokenizer
- Simple Tokenizer - Regex Based (Experimental)
📖 For more details, refere to sadedegel.ai
Install sadedeGel
- Operating system: macOS / OS X · Linux · Windows (Cygwin, MinGW, Visual Studio)
- Python version: 3.6+ (only 64 bit)
- Package managers: pip
pip
Using pip, sadedeGel releases are available as source packages and binary wheels.
pip install sadedegel
When using pip it is generally recommended to install packages in a virtual environment to avoid modifying system state:
python -m venv .env
source .env/bin/activate
pip install sadedegel
Quickstart with SadedeGel
To load SadedeGel, use sadedegel.load()
from sadedegel import Doc
from sadedegel.dataset import load_raw_corpus
from sadedegel.summarize import Rouge1Summarizer
raw = load_raw_corpus()
d = Doc(next(raw))
summarizer = Rouge1Summarizer()
summarizer(d, k=5)
To use our ML based sentence boundary detector
from sadedegel import Doc
doc = ("Bilişim sektörü, günlük devrimlerin yaşandığı ve hızına yetişilemeyen dev bir alan haline geleli uzun bir zaman olmadı. Günümüz bilgisayarlarının tarihi, yarım asırı yeni tamamlarken; yaşanan gelişmeler çok "
"daha büyük ölçekte. Türkiye de bu gelişmelere 1960 yılında Karayolları Umum Müdürlüğü (şimdiki Karayolları Genel Müdürlüğü) için IBM’den satın aldığı ilk bilgisayarıyla dahil oldu. IBM 650 Model I adını taşıyan bilgisayarın "
"satın alınma amacı ise yol yapımında gereken hesaplamaların daha hızlı yapılmasıydı. Türkiye’nin ilk bilgisayar destekli karayolu olan 63 km uzunluğundaki Polatlı - Sivrihisar yolu için yapılan hesaplamalar IBM 650 ile 1 saatte yapıldı. "
"Daha öncesinde 3 - 4 ayı bulan hesaplamaların 1 saate inmesi; teknolojinin, ekonomik ve toplumsal dönüşüme büyük etkide bulunacağının habercisiydi.")
Doc(doc).sents
['Bilişim sektörü, günlük devrimlerin yaşandığı ve hızına yetişilemeyen dev bir alan haline geleli uzun bir zaman olmadı.',
'Günümüz bilgisayarlarının tarihi, yarım asırı yeni tamamlarken; yaşanan gelişmeler çok daha büyük ölçekte.',
'Türkiye de bu gelişmelere 1960 yılında Karayolları Umum Müdürlüğü (şimdiki Karayolları Genel Müdürlüğü) için IBM’den satın aldığı ilk bilgisayarıyla dahil oldu.',
'IBM 650 Model I adını taşıyan bilgisayarın satın alınma amacı ise yol yapımında gereken hesaplamaların daha hızlı yapılmasıydı.',
'Türkiye’nin ilk bilgisayar destekli karayolu olan 63 km uzunluğundaki Polatlı - Sivrihisar yolu için yapılan hesaplamalar IBM 650 ile 1 saatte yapıldı.',
'Daha öncesinde 3 - 4 ayı bulan hesaplamaların 1 saate inmesi; teknolojinin, ekonomik ve toplumsal dönüşüme büyük etkide bulunacağının habercisiydi.']
SadedeGel Server
In order to integrate with your applications we provide a quick summarizer server with sadedeGel.
python3 -m sadedegel.server
SadedeGel Server on Heroku
SadedeGel Server is hosted on free tier of Heroku cloud services.
PyLint, Flake8 and Bandit
sadedeGel utilized pylint for static code analysis, flake8 for code styling and bandit for code security check.
To run all tests
make lint
Run tests
sadedeGel comes with an extensive test suite. In order to run the
tests, you'll usually want to clone the repository and build sadedeGel from source.
This will also install the required development dependencies and test utilities
defined in the requirements.txt
.
Alternatively, you can find out where sadedeGel is installed and run pytest
on
that directory. Don't forget to also install the test utilities via sadedeGel's
requirements.txt
:
make test
Youtube Channel
Some videos from sadedeGel YouTube Channel
SkyLab YTU Webinar Playlist
References
Our Community Contributors
We would like to thank our community contributors for their bug/enhancement requests and questions to make sadedeGel better everyday
Software Engineering
-
Special thanks to spaCy project for their work in showing us the way to implement a proper python module rather than merely explaining it.
- We have borrowed many document and style related stuff from their code base :smile:
-
There are a few free-tier service providers we need to thank:
- GitHub for
- Hosting our projects.
- Making it possible to collobrate easily.
- Automating our SLM via Github Actions
- Google Cloud Google Storage Service for providing low cost storage buckets making it possible to store
sadedegel.dataset.extended
data. - Heroku for hosting sadedeGel Server in their free tier dynos.
- CodeCov for allowing us to transparently share our test coverage
- PyPI for allowing us to share sadedegel with you.
- binder for
- Allowing us to share our example notebooks
- Hosting our learn by example boxes in sadedegel.ai
- GitHub for
Machine Learning (ML), Deep Learning (DL) and Natural Language Processing (NLP)
-
Resources on Extractive Text Summarization:
- Leveraging BERT for Extractive Text Summarization on Lectures by Derek Miller
- Fine-tune BERT for Extractive Summarization by Yang Liu
-
Other NLP related references
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for sadedegel-0.15.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9926e999394fb69cb87460d07120bd2cb439bb5222211f4b42290b00a39a545b |
|
MD5 | 655d78411a709869561cd719b4f04989 |
|
BLAKE2b-256 | f8fb20f77518b03aec15ec28a7465468a868f5817648630b8680e1bb058a8dad |