Digital Humanities Utilities

Python 3.6+ package containing various utilities relevant in the field of digital humanities.

$ pip install dh-utils

Unicode utilities

Convert Greek beta code to unicode

>>> from dh_utils.unicode import beta2uni
>>> beta2uni('lo/gos')
'λόγος'

This is a wrapper of the CLTK converter. We used this converter to also create inverse:

>>> from dh_utils.unicode import uni2beta
>>> uni2beta('λόγος')
'lo/gos'

NB: Since cltk and its dependency nltk are relatively large, cltk is added as an optional dependency. To use the beta2uni converter, either install cltk separately using pip install cltk or install dh-utils including this optional depency with pip install dh-utils[betacode].

Decompose a unicode string

>>> u.decompose('λόγος')
λ U+03bb GREEK SMALL LETTER LAMDA
ο U+03bf GREEK SMALL LETTER OMICRON
́ U+0301 COMBINING ACUTE ACCENT
γ U+03b3 GREEK SMALL LETTER GAMMA
ο U+03bf GREEK SMALL LETTER OMICRON
ς U+03c2 GREEK SMALL LETTER FINAL SIGMA

TEI utilities

Convert markdown to TEI

A basic converter from markdown to TEI has been added. It will convert a markdown file like:

Some paragraph block

> A blockquote

1. An
2. Ordered
3. List

Another paragraph block with _italics_ and __bold__, and:

* An
* Unordered
* List

using a Python snippet like

>>> from dh_utils.tei import md2tei
>>> with open('file.md') as f:
>>>    md = md2tei(f.read())

to the following TEI XML:

<p>Some paragraph block</p>
<quote>
  <p>A blockquote</p>
</quote>
<list rend="numbered">
  <item>An</item>
  <item>Ordered</item>
  <item>List</item>
</list>
<p>Another paragraph block with <hi rend="italic">italics</hi> and <hi rend="bold">bold</hi>, and:</p>
<list rend="bulleted">
  <item>An</item>
  <item>Unordered</item>
  <item>List</item>
</list>

The function md2tei is syntactic sugar for the markdown extension ToTEI, which can be used in combination with other extensions as follows:

>>> from markdown import markdown
>>> from dh_utils.tei import ToTEI
>>> markdown('some text', extensions=[ToTEI()]) # Other extensions can be added to this list

The extension ToTEI in turn exists solely of the postprocessor TEIPostprocessor. It has priority 0, which usually means that it will run after all other postprocessors have finished. If any other behaviour or prioritization is required, the processor TEIPostprocessor can also be directly imported (from dh_utils.tei import TEIPostprocessor) and used in a custom markdown extension.

Tag languages

Tag languages in a given string based on its script:

>>> from dh_utils.tei import tag_script
>>> tag_script('A line contaning the hebrew אגוז מלך inline', 'Hebr')
'A line contaning the hebrew <foreign xml:lang="he-Hebr">אגוז מלך</foreign> inline'

It is also possible to tag a given language based on its script in a TEI XML document (NB: file will be overwritten!):

>>> from dh_utils.tei import tag_script_from_file
>>> tag_script_from_file('path/to/file.xml', 'Arab')

The available scripts are stored in dh_utils.tei.AVAILABLE_SCRIPTS and are enumerated below:

>>> from dh_utils.tei import AVAILABLE_SCRIPTS
>>> AVAILABLE_SCRIPTS
['Arab', 'Copt', 'Hebr', 'Latn', 'Cyrl']

Default language-script codes are used to tag the scripts (stored in DEFAULT_LCS), which can be adjusted using the language_code keyword argument:

>>> t.tag_script_from_file('path/to/file.xml', 'Cyrl', language_code = 'ov-Cyrs')

Refsdecl generator

To generate refsdecl elements, the generator can be used to create etree xml elements:

from dh_utils.tei import refsdecl_generator

refs_decl = refsdecl_generator.generate_for_file("./path/to/file")
refs_decls = refsdecl_generator.generate_for_path("./path/to/files")

It can also be used trough the command line interface:

python -m dh_utils.tei.refsdecl_generator [--update] [PATH]

By default, it does not update the file but outputs the refsdecl xml to the terminal. If the --update flag is given, the file is updated with the generated refsdecl.

MyCapytain-compatilble critical apparatus

The Python API MyCapytain only serves the main text of a CTS structured text version, and does not support stand-off annotation, bibliographies, critical apparati, etc. To overcome the last problem, we have developed a script that generates a separate text version of the critical apparatus that can be served through MyCapytain. Brill's Scholarly Editions uses these separate text versions, which can be displayed in parallel.

The following snippet creates such a critapp file from textgroup.work.edition-extension.xml located in path/to/data/textgroup/work and saves it as textgroup.work.edition-appcrit1.xml

>>> from dh_utils.tei import crit_app as ca
>>> data_dir = "path/to/data/textgroup/work"
>>> filename = "textgroup.work.edition-extension.xml"
>>> ca_ext = "appcrit1" # Or any other extension
>>> ca.create(filename, ca_ext, data_dir)

If a file contains multiple critical apparati, these can be distinguished using /listApp[@type], e.g.:

<listApp type="superior">
  <app/>
  <app/>
  ...
</listApp>
<listApp type="inferior">
  <app/>
  <app/>
  ...
</listApp>

Using the above snippet will combine these apparati into one file. If these should be conerted to separate files, one can pass an additional argument app_type to ca.create (e.g., ca.create(filename, ca_ext, data_dir app_type="superior")) to convert an apparatus separately.