jsonl-archive-splitter 0.1.2

CLI tool to split JSONL archive files by Jalali boundary date.

JSONL Archive Splitter

A production-ready Python CLI tool for splitting large JSONL archive files based on a Jalali (Persian) boundary date and configurable maximum file size.

---

✨ Features

• Split JSONL files based on publish_ts
• Supports Jalali (Shamsi) boundary dates
• Automatic file rotation by maximum size
• Line-by-line processing (memory efficient)
• Modular architecture (CLI / Core / Utils separation)
• Logging-based execution (no noisy prints)
• Ready for packaging and future PyPI publishing

---

📦 Installation

using source code (GitHub repository):

git clone https://github.com/ngsaeed/jsonl-archive-splitter.git
cd jsonl-archive-splitter

using pip:

pip install jsonl-archive-splitter

requirements.txt:

jdatetime

---

🚀 Usage

Run from the project root directory:

jsonl-archive-splitter

Or use arguments to best fit your requirements:

jsonl-archive-splitter --src-dir D:\\jsonl-archive-splitter\\src --before-dir D:\\jsonl-archive-splitter\\output\\part1 --after-dir D:\\jsonl-archive-splitter\\output\\part2 --boundary-date 1404-07-30 --start 1 --end 10 --max-size 400

Expected Source File Pattern (Default)

By default, the tool looks for files with this naming pattern:

archive_messages_<index>.jsonl

Example:

archive_messages_1.jsonl
archive_messages_2.jsonl
archive_messages_3.jsonl

you can set prefix or postfix for making the pattern that best fits to your archive. for example your archive is like messages_<index>.txt so:

jsonl-archive-splitter --src-prefix 'messages_' --src-postfix '.txt'

This will look for files:
messages_1.txt
...
messages_10.txt

By default, it processes only files between index 1 and 10:

archive_messages_1.jsonl
...
archive_messages_10.jsonl

But you can set it to process files between whatever indices you want. for example:

jsonl-archive-splitter --start 201 --end 400

will look for following files:
archive_messages_201.jsonl
...
archive_messages_400.jsonl

The boundary date must be in Jalali format: YYYY-MM-DD for example:

jsonl-archive-splitter --boundary-date '1404-12-04'

---

🧾 CLI Arguments

Argument Description

---

--src-dir Directory containing input JSONL files (default: D:\jsonl-archive-splitter\src)
--before-dir Output directory to store files for messages up to boundary date (default: D:\jsonl-archive-splitter\output\part1)
--after-dir Output directory to store files for messages after boundary date (default: D:\jsonl-archive-splitter\output\part2)
--boundary-date Jalali date in format YYYY-MM-DD (default 1404-07-30)
--start Start file index (default: 1)
--end End file index (default: 10)
--max-size Max size per output file in MB (default: 400)
--src-prefix Source files prefix (default: archive_messages_)
--src-postfix Output files format or postfix (default: .jsonl)
--out-prefix Output files prefix (default: archive_messages_)

---

🧠 How It Works

1. Converts Jalali boundary date to Gregorian.
2. Computes timestamp for 23:59:59 of boundary date.
3. Iterates over JSONL files sequentially.
4. Routes messages based on publish_ts.
5. Rotates output files automatically when size limit is reached.

---

🔮 Possible Future Improvements

• Add unit tests
• Add tqdm progress bar
• Add dry-run mode

---

📄 License

MIT License

---

👤 Author

Saeed Naghibi -- Front-End Software Engineer \ Algorithm Specialist
GitHub: https://github.com/ngsaeed
LinkedIn: https://www.linkedin.com/in/ngsaeed/