A package for dealing with the openITI corpus
Project description
openiti
This is a first attempt to create a Python library that combines all often-used code in the OpenITI project. Full documentation and deescription can be found here: https://openiti.readthedocs.io/
Installation
pip install OpenITI
Alternatively, you might need to use pip3 install OpenITI
or python -m pip install OpenITI
.
Change log:
v.0.1.3:
- new_books.convert.`: add converters for ALCorpus and Ptolemaeus texts
new_books.convert.helper.html2md
: tweaks to import of options + small tweakshelper.ara
: Stop ar_cnt_file from raising exception if book misses splitter; instead, print warninghelper.funcs
:- fix bug in
get_all_text_files_in_folder
function: missing periods in regex. - improve missing splitter message
- use
ara.normalize_ara_light
function instead ofara.normalize_ara_extra_light
intext_cleaner
function
- fix bug in
helper.uri
:- make it possible to pass a specific
version_fp
to thecheck_token_count
function; before, that function generated that path from the URI, but this created problems when files were not stored in the standard OpenITI folders. - add
find_latest
parameter incheck_token_count
function; if ``False, the function will count the tokens in the specific
version_fp` provided; if `True, the script will count tokens in the file with the most advanced extension (.mARkdown > .completed > .inProgress > [no extension])
- make it possible to pass a specific
helper.yml
:- make it possible to pass a specific
yml_fp
to theymlToDic
function, so that the script can print the path (if provided) for signalling empty yml files. - Include possibility that yml key ends with more than one colon
readYML
: add exception message when yml file could not be read.
- make it possible to pass a specific
v.0.1.2:
openiti.helper.funcs
: Fixed bug in report_missing_numbers function.openiti.new_books.convert
: added ShamAY converter and small updates to other shamela converters.
v.0.1.1:
openiti.helper.funcs
: Added get_all_text_files_in_folder() generatoropeniti.helper.uri
: Fix bug innew_yml
function (URI used to have ".yml" in it)openiti.new_books.convert.shamela_converter.py
: Improved formatting of the text and notes and added support for shamela collections in which the .mdb files contain more than one book.openiti.new_books.convert.tei_converter_Thielen
,new_books/convert/helper/html2md_Thielen
: added new converter for TEI files provided by Jan Thielen.new_books/convert/tei_converter_generic.py
,new_books/convert/helper/html2md.py
: Add the possibility to pass options to themarkdownify
function
v.0.1.0:
openiti.helper.yml
: add support for empty lines and bullet lists in multiline valuesopeniti.new_books.convert.shamela_converter
: fix bugs in shamela converter
v.0.0.9.post1 (patch):
openiti.helper.ara
: fix bug in regex compilation
v.0.0.9:
openiti.new_books.convert
: check and update all convertersopeniti.helper.ara
: make counting characters in editorial sections optional (default: include Arabic characters in editorial sections)openiti.helper.yml
: add custom error messages for broken and empty yml filesopeniti.git.git_util
: add git utilities class, withcommit
method
v.0.0.8:
openiti.new_books.add.add_books
: fix import bugopeniti.new_books.convert
: add converter for Noorlib html files
v.0.0.7:
openiti.git.get_issues
: change authentication from username/password to GitHub tokenopeniti.helper.ara
: add function to normalize composite Arabic charactersopeniti.helper.uri
: move functions for adding texts to the corpus to a new module,openiti.new_books.add.add_books
openiti.helper.uri
: fix bug in the character count function (did not work if execute==True)openiti.new_books.convert
: restructured folder and moved helper functions into a new subfolder calledhelper
openiti.new_books.convert.generic_converter
:- reordered the main
convert_file
function and added inline documentation - made
convert_files_in_folder
function more flexible
- reordered the main
openiti.new_books.convert
: added generic converters for shamela libraries, html and tei xml files, and custom converters for eShia and GRAR librariesopeniti.new_books.convert.shamela_converter
openiti.new_books.convert.html_converter_generic
openiti.new_books.convert.html_converter_eShia
openiti.new_books.convert.tei_converter_generic
openiti.new_books.convert.tei_converter_GRAR
openiti.new_books.convert.helper
: added helper functions for the new converters:openiti.new_books.convert.helper.html2md_eShia
openiti.new_books.convert.helper.html2md_GRAR
openiti.new_books.convert.helper.tei2md
openiti.new_books.convert.helper.bok
v.0.0.6:
openiti.helper.uri
: use both Arabic character and token count in yml filesopeniti.helper.uri
: add support for paths to files that are not in 25-years repos (e.g., for release)openiti.helper.uri
: fix bugs- added Sphinx documentation
v.0.0.5:
openiti.helper.funcs
: added Arabic token count functionopeniti.helper.uri
: use Arabic token count instead of Arabic character count for yml file revision. Also, revise token count for every version yml file instead of only for version yml files that do not contain a count.
v.0.0.4:
openiti.helper.uri
: removed the restriction on the use of digits in book titlesopeniti.helper.uri
: added a check for empty yml filesopeniti.helper.yml
: added documentation and doctestsopeniti.helper.yml
: added check for empty yml files + changed splitting of yml files so that even unindented multi-line values can be correctly parsed.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
openiti-0.1.3.tar.gz
(124.9 kB
view hashes)
Built Distribution
openiti-0.1.3-py3-none-any.whl
(197.0 kB
view hashes)