cwb-vrt

cwb-vrt is a Python 3 module and command line interface for processing VRT files.

• Installation
• VRT files
  • VRT files for CQPweb
• Using the CLI

Installation

Requires Python 3.10+.

via PyPI

pip install cwb-vrt

via git (latest, unreleased changes)

pip install git+https://github.com/ausgerechnet/cwb-vrt.git

VRT files

VRT files are XML-like files containing verticalised text and are used as an import (and export) format of the IMS Open Corpus Workbench (CWB). The CWB distinguishes positional attributes (p-atts) on a token-level, which are stored in tab-separated lines, and structural attributes (s-atts) stored in XML elements (i.e. matching pairs of start and end tags).

<?xml version="1.0" encoding="ISO-8859-1" standalone="yes" ?>
<!-- A Thrilling Experience -->
<story num="4" title="A Thrilling Experience">
<p>
<s>
Tick	NN	tick
.	SENT	.
</s>
<s>
A	DT	a
clock	NN	clock
.	SENT	.
</s>
<s>
Tick	VB	tick
,	,	,
tick	VB	tick
.	SENT	.
</s>
</p>
...
</story>
...

The VRT file above contains three p-atts: by default, the first or primary layer is called word — the other p-atts here could e.g. be named pos and lemma. Note that the names of p-atts are usually not explicitly encoded in VRT files.

There are also three s-atts encoded in XML-elements: story, p, and s. story has two attribute-value pairs:

<story num="4" title="A Thrilling Experience">

cwb-vrt refers to the name of the XML element (e.g. story) usually as "level" of the s-att and stores it alongside the other key-value pairs in a dictionary. Note that in the CWB, each attribute is stored separately (here: story_num and story_title) with annotation (story itself is not encoded as an s-att). p and s do not have any annotation.

Unlike well-formed XML, s-attribute regions in a VRT file don't have to nest properly — they may overlap (also called "crossing" or "interleaved" annotations). For example, a <s> spanning a <p> boundary is valid VRT but not valid XML: <p>...<s>...</p><p>...</s>...</p>.

VRT files for CQPweb

Not all VRT files accepted by the CWB can be used for CQPweb:

• there has to be an XML-element called "text" with unique IDs <text id="...">
• only a relatively small number of <text>s are possible (~ 10,000,000)
• meta data stored in <text>-attributes can be used for subcorpus creation and restricted queries, but they have to be marked as categorical in CQPweb — and this only works if all values are valid MySQL-identifiers

Using the CLI

Installing cwb-vrt puts a single cwb-vrt command on your PATH, with one subcommand per tool. Run cwb-vrt --help for the full list, or cwb-vrt <subcommand> --help for a subcommand's options.

$ cwb-vrt --help
usage: cwb-vrt [-h] [-v]
               {bio-repair,convert,cqpweb,dedup,encode,group,merge,meta,remove}
               ...

cwb-vrt: tools for processing VRT files

positional arguments:
  {bio-repair,convert,cqpweb,dedup,encode,group,merge,meta,remove}
    bio-repair          repair (potentially invalid) BIOES token tagging into
                        valid BIO tagging
    convert             render VRT file(s) as HTML (or plain text)
    cqpweb              make VRT file compatible with CQPweb
    dedup               find duplicate regions, or remove them from the VRT
    encode              create CWB import script from VRT file
    group               conflate texts according to meta data into cohorts
    merge               merge several VRT files of cohorts into one VRT
    meta                create TSV table of meta data stored in s-atts
    remove              remove structural and/or positional attributes from
                        VRT file

options:
  -h, --help            show this help message and exit
  -v, --version         output version information and exit

cwb-vrt bio-repair: repair (potentially invalid) BIOES token tagging into valid BIO tagging, storing it as an s-attribute

cwb-vrt bio-repair ner-bioes.vrt.gz --col 2 --level ner

cwb-vrt convert: render VRT file(s) as HTML (or plain text), optionally highlighting spans annotated with a given s-attribute (e.g. <sub> for sensitive data)

cwb-vrt convert tagesschau-mini-cqpweb.vrt.gz --highlight sub

cwb-vrt cqpweb: make VRT file compatible with CQPweb

cwb-vrt cqpweb tagesschau-mini.vrt.gz --level article

cwb-vrt dedup: find regions enclosed by level that are duplicates (--mode report, default: writes a TSV table), or remove them directly (--mode deduplicate: writes a cleaned copy of the VRT)

cwb-vrt dedup tagesschau-mini.vrt.gz --level s
cwb-vrt dedup tagesschau-mini.vrt.gz --level s --mode deduplicate

cwb-vrt encode: create CWB import script from VRT file

cwb-vrt encode tagesschau-mini.vrt.gz

cwb-vrt group: conflate texts according to meta data into cohorts

cwb-vrt group -m tagesschau-mini.vrt.gz -c month rubrik --level-old article --level-new article

cwb-vrt merge: merge several VRT files of cohorts into one VRT

cwb-vrt merge 'tagesschau-mini-cohorts/*.vrt.gz' --name TAGESSCHAU

cwb-vrt meta: create TSV table of meta data stored in s-atts

cwb-vrt meta tagesschau-mini.vrt.gz --level article

cwb-vrt remove: remove structural and/or positional attributes from VRT file, or just selected keys from a structural attribute's annotation

cwb-vrt remove tagesschau-mini.vrt.gz --s_atts p --p_atts 1 --s_att_keys article:fname