Skip to main content

Hazm

Tests PyPI - Downloads GitHub

Python library for digesting Persian text.

  • Text cleaning
  • Sentence and word tokenizer
  • Word lemmatizer
  • POS tagger
  • Shallow parser
  • Dependency parser
  • Interfaces for Persian corpora
  • NLTK compatible
  • Python 3.8, 3.9, 3.10 and 3.11 support

Documentation

Visit https://roshan-ai.ir/hazm/docs to view the full documentation.

Modules accuracy

Module name accuracy
Lemmatizer 89.9%
Chunker 93.4% download pre-trained model
POSTagger 97.2% download pre-trained model
POSTagger(Universal) 98.8% download pre-trained model
DependencyParser 97.1% download pre-trained model

Installation

The latest stable version of Hazm can be installed through pip:

pip install hazm

But for testing or using Hazm with the latest updates you may use:

pip install https://github.com/roshan-research/hazm/archive/master.zip --upgrade

Usage

>>> from hazm import *

>>> normalizer = Normalizer()
>>> normalizer.normalize('اصلاح نويسه ها و استفاده از نیم‌فاصله پردازش را آسان مي كند')
'اصلاح نویسه‌ها و استفاده از نیم‌فاصله پردازش را آسان می‌کند'

>>> sent_tokenize('ما هم برای وصل کردن آمدیم! ولی برای پردازش، جدا بهتر نیست؟')
['ما هم برای وصل کردن آمدیم!', 'ولی برای پردازش، جدا بهتر نیست؟']
>>> word_tokenize('ولی برای پردازش، جدا بهتر نیست؟')
['ولی', 'برای', 'پردازش', '،', 'جدا', 'بهتر', 'نیست', '؟']

>>> stemmer = Stemmer()
>>> stemmer.stem('کتاب‌ها')
'کتاب'
>>> lemmatizer = Lemmatizer()
>>> lemmatizer.lemmatize('می‌روم')
'رفت#رو'

>>> tagger = POSTagger(model='resources/pos_tagger.model')
>>> tagger.tag(word_tokenize('ما بسیار کتاب می‌خوانیم'))
[('ما', 'PRO'), ('بسیار', 'ADV'), ('کتاب', 'N'), ('می‌خوانیم', 'V')]

>>> chunker = Chunker(model='resources/chunker.model')
>>> tagged = tagger.tag(word_tokenize('کتاب خواندن را دوست داریم'))
>>> tree2brackets(chunker.parse(tagged))
'[کتاب خواندن NP] [را POSTP] [دوست داریم VP]'

>>> parser = DependencyParser(tagger=tagger, lemmatizer=lemmatizer)
>>> parser.parse(word_tokenize('زنگ‌ها برای که به صدا درمی‌آید؟'))
<DependencyGraph with 8 nodes>

Extensions

Note: These are not official versions of hazm, not uptodate on functionality and are not supported by Roshan.

  • JHazm: A Java port of Hazm
  • NHazm: A C# port of Hazm

Contribution

We welcome and appreciate any contributions to this repo, such as bug reports, feature requests, code improvements, documentation updates, etc. Please follow the Contribution guideline when contributing. You can open an issue, fork the repo, write your code, create a pull request and wait for a review and feedback. Thank you for your interest and support in this repo!

Thanks

Code contributores

Alt

Others

  • Thanks to Virastyar project for providing the persian word list.

Star History Chart

Metadata

Release files for hazm 0.9.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for hazm 0.9.1
File Size Uploaded Provenance Actions
hazm-0.9.1.tar.gz 333.7 kB Without provenance Details for hazm-0.9.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for hazm 0.9.1
File Interpreter ABI Platform Actions
hazm-0.9.1-py3-none-any.whl Python 3 none any Details for hazm-0.9.1-py3-none-any.whl

Total release size: 683.4 kB

Release files / hazm-0.9.1.tar.gz

Download URL hazm-0.9.1.tar.gz
Size 333.7 kB
Tags Source
SHA-256 checksum
How to use checksums
be52ad5c46c09186ebeaf6169f1638afceb4f9330211fd0dacd25736a6ac3d7c
BLAKE2b-256 checksum
How to use checksums
1f6ed8cedea269e5e72cb863ae5d7a53d1e09ce5c963346edd20893f97a1e9e6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/1.5.1 CPython/3.8.0 Windows/10

Release files / hazm-0.9.1-py3-none-any.whl

Download URL hazm-0.9.1-py3-none-any.whl
Size 349.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
eb75ab0c45378614b261e1db33cea5c836a0a7fd3d7bc93e419d0651313bbfca
BLAKE2b-256 checksum
How to use checksums
c82d447005ee30162d36da7317ec6d115f8a531bba66a6c4ca0daf1aaa77b21c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via poetry/1.5.1 CPython/3.8.0 Windows/10

Security for hazm 0.9.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page