python-iransystem

Nearly all old Iranian programs (especially programs from the golden DOS era) and a great number of new enterprise programs use the IranSystem encoding. Here is a Python module for easy reading and writing IranSystem encoded files.

Installing

The module is published on PyPI. In a project managed with uv, add it as a dependency:

uv add iransystem

To install it into the current environment instead, use uv pip or pip:

uv pip install iransystem
pip install iransystem

To try it without installing anything, let uv run a script with the module available:

uv run --with iransystem script.py

To get the latest development version, install it from the repository:

pip install git+https://git.sr.ht/~gurban/python-iransystem

Using the module

The module needs Python 3.10 or newer. Importing it registers the iransystem and iransystem_visual codecs (iran_system, iran-system, iransystem-visual and iran_system_visual work too):

import iransystem

with open("document.txt", encoding="iransystem") as f:
    text = f.read()

The module also offers encode() and decode(), which convert a whole string like str.encode() and bytes.decode() do. They use the iransystem codec, or iransystem_visual with visual=True:

from iransystem import decode, encode

data = encode("سلام دنیا")
decode(data)  # "سلام دنیا"
decode(data, visual=True)  # "ایند مالس"

IranSystem stores text in visual order (each line laid out left to right as it appears on screen), so Persian text is stored backwards. The iransystem codec converts each line to logical order while decoding (and back while encoding). It keeps Latin words and numbers in their original order, along with email addresses, URLs, DOS paths and words such as C++ or 50%, and mirrors brackets. Punctuation that ends a sentence stays with the Persian text. If you need the raw visual order, use the iransystem_visual codec.

The same reordering is available as iransystem.flip(), for text you already have in visual order (or want in visual order). It converts either way, so flip(flip(text)) == text:

from iransystem import flip

flip("Workbench 2.0 مالس")  # "سلام Workbench 2.0"
flip("سلام Workbench 2.0")  # "Workbench 2.0 مالس"

IranSystem also stores each letter in its contextual form (initial, medial, final or isolated). Where those forms show two letters are not joined but Unicode would join them, the iransystem codec adds a ZERO WIDTH NON-JOINER (as in «می‌کند» or «خانه‌ها»). When encoding, it picks the form of each letter from its neighbours and respects ZERO WIDTH NON-JOINER and ZERO WIDTH JOINER.

IranSystem has no codes for the two joiners themselves. When encoding, they only choose the forms of the letters around them and are then dropped, so "ب‍" encodes as a BEH that joins the next letter, and a joiner on its own encodes to nothing. When decoding, the iransystem codec adds a ZERO WIDTH NON-JOINER back only where the stored forms show a break the renderer wouldn't make, so a non-joiner that changed no form (after ALEF, for example) is lost, and so is every ZERO WIDTH JOINER.

Text is converted line by line

Both codecs (and flip()) work on one line at a time. A line ends at LF or CR, so CR LF ends one line; no other character (such as form feed or U+2028 LINE SEPARATOR) ends a line. Each line is reordered on its own, and the form of a letter depends only on its neighbours in that line. What this means depends on how the text is converted:

• str.encode() and bytes.decode() (and encode() and decode()) see the whole text at once, so they are always correct.
• Reading with iransystem (through open(), an incremental decoder or a StreamReader) holds back an incomplete last line until the rest of it arrives or the input ends, so the bytes can come in any pieces. Reading with iransystem_visual maps each byte on its own, so the same is true.
• Writing (through open(), an incremental encoder or a StreamWriter) converts each piece on its own, so each write() must hold whole lines (see Write whole lines).

Examples

Example 1: Read a text file

import iransystem

with open("document.txt", encoding="iransystem") as f:
    for line in f:
        print(line, end="")

Example 2: Write a text file

IranSystem files usually come from DOS, so write DOS line endings:

import iransystem

with open("document.txt", "w", encoding="iransystem", newline="\r\n") as f:
    f.write("سلام دنیا\n")

Example 3: Read a Paradox DBF file

import dbfread
import iransystem

# Read an Iranian Tamin Ejtema-ee's Paradox DBF file
table = dbfread.DBF("DSKWOR00.dbf", encoding="iransystem")

for record in table:
    print(f"{record['DSW_FNAME']} {record['DSW_LNAME']}")

Problems

Write whole lines

When writing with the iransystem or iransystem_visual codec (through open(), codecs.open() or a StreamWriter), each write() call is encoded separately: the iransystem codec converts it to visual order on its own, and both codecs pick letter forms without seeing the text of the next call. So write whole lines at a time. For example, print("سلام", "دنیا", file=f) writes its arguments one by one and stores them in the wrong order, and f.write("سل"); f.write("ام") stores LAM and ALEF as separate letters instead of the LAM-ALEF ligature of «سلام». Join the text first: f.write("سلام دنیا\n").

TODO

• Report encoding errors at their position in the input. Errors are raised after the text is shaped and reordered, so "سلام €".encode("iransystem") reports position 0, not 5, and error handlers see the reordered text.
• Return the input length as consumed when encoding, not the length of the shaped text ("لا\u200c" reports 1 instead of 3).
• Reject private use characters U+F78D to U+F7FE in the input. They are used internally for shaped letters, so "\uf792" encodes silently as BEH.
• Add ZERO WIDTH NON-JOINERs when decoding with iransystem_visual too. Without them it loses which letters are shown apart, and encoding the text again joins them.
• Keep symbols between a Latin and a Persian word with the Persian word: in «سلام GNU/لینوکس» the slash is stored on the GNU side.
• Encode across write() calls, so text doesn't have to be written in whole lines (see Write whole lines).

Development

The project uses uv. uv sync creates a virtual environment in .venv with the module and the development tools (ruff), as pinned in uv.lock; uv run runs a command in it, syncing it first if needed. From the repository root:

uv sync                         # set up the environment
uv run python -m unittest       # run the tests
uv run ruff check .             # lint
uv run ruff format .            # format the code
uv run samples/decode-text.py   # decode the sample file
uv build                        # build the wheel and sdist into dist/

The version is __version__ in iransystem.py; pyproject.toml reads it from there.

License

Copyright © 2020 Ghorban M. Tavakoly

This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see https://www.gnu.org/licenses/.