livivo-converter

Get a new metadata source into LIVIVO by writing a mapping, not code.

livivo-converter reads a JSONL export, one record per line, and writes the XML that the LIVIVO import expects. What makes a given source work is not Python: it is a single YAML mapping file saying which LIVIVO field gets which value, which records to leave out and which count as withdrawn. The mapping format is a small DSL of its own, with paths into the record, format strings and conditions, and it carries the whole translation. So bringing a new source into LIVIVO is one YAML file, and the converter stays untouched.

$ livivo-converter export.jsonl livivo.xml --source biorxiv
INFO - Wrote livivo.xml.gz: 1284 documents, 3 delete entries, 0 skipped

The mappings that ship with the package cover preprint servers, and their input is the kind of file produced by pixls-preprints, which harvests bioRxiv, medRxiv and OSF. The two together form one pipeline: harvest to JSONL, convert to LIVIVO XML. Nothing in the converter itself is specific to preprints: any source you can export as JSONL can be mapped the same way.

Why use it

• New sources without new code. A source is a YAML mapping file. Adding one is a change to configuration, not to the converter.
• Filtering is part of the mapping. A skip condition decides what never gets imported. The OSF mapping uses it as a life-science relevance filter, keeping some preprint providers wholesale and others only for medical and life-science subjects.
• Withdrawn preprints are handled, not dropped. Records matching a mapping's withdrawal condition become delete entries, which tell LIVIVO to remove a record it has already imported.
• Mistakes surface early. Mappings are schema-checked when loaded, and a missing input file or an unknown source ends the run before any input is read, rather than part way through a large export.
• Output the importer accepts. Field text is cleaned on the way out, with control characters removed and line breaks replaced by the marker the import understands.
• Compressed by default. Output is gzipped unless you ask for plain XML.

Requirements

• Python 3.13 or newer (see requires-python in pyproject.toml)
• uv to install from source or to work on the converter

Installation

Run the package as a module from a source checkout:

git clone https://codeberg.org/pixls/livivo-converter.git
cd livivo-converter
uv sync
uv run python -m livivo_converter export.jsonl livivo.xml --source biorxiv

Installing the package instead provides a livivo-converter command, which takes the same arguments and brings its mappings with it, so there is nothing else to set up:

uv pip install livivo-converter
livivo-converter export.jsonl livivo.xml --source biorxiv

uv tool install livivo-converter puts that command on your path without touching the environment you are in, and for a single conversion uvx livivo-converter ... fetches and runs it in one go, installing nothing at all.

Usage

Give the converter an input file, an output file and the source to convert:

livivo-converter export.jsonl livivo.xml --source biorxiv

The output is gzip-compressed and .gz is appended to the name you passed, so the command above writes livivo.xml.gz. Pass --no-zip for plain XML:

livivo-converter export.jsonl livivo.xml --source osf --no-zip
# -> livivo.xml

From a source checkout the same run is uv run livivo-converter ..., which reaches the same entry point as the module form above.

The source name is the name of its mapping file. Three ship with the package: biorxiv, medrxiv and osf. An unknown source is rejected before any input is read.

Options

Option	Description
input_file	Required. File to convert, one JSON record per line (JSONL)
output_file	Required. XML file to write, with .gz appended unless --no-zip is given
--source	Required. Source to convert, named after its mapping file
--no-zip	Write uncompressed XML instead of gzip-compressed XML
-f, --force	Overwrite the output file if it exists
-v, --verbose	Log at DEBUG level instead of INFO

An existing output file is not overwritten. The run stops before any input is read unless -f is given, and for a compressed run it is the .gz file that is checked, since that is the one being written.

A run ends with one line counting what was written, deleted and skipped. Which records those were is DEBUG detail, since a relevance filter is meant to drop the greater part of an export. Pass -v and each skipped record is reported with the values its skip rule tested, which is usually enough to see why it was left out, and each converted record with the fields as they were built.

Mapping files

Each source is described by one YAML file in livivo_converter/mappings/. A mapping says which LIVIVO field gets which value, which records to leave out and which count as withdrawn:

source: biorxiv

settings:
  delimiters: # split these input fields into lists first
    authors: ";"
  skip_values: # values that count as no value
    - "NA"

fields:
  DATABASE:
    val: PIXLS # a constant
  TITLE:
    map: title # copied from the record
  PUBLYEAR:
    map: date
    spec: T[:4] # sliced on the way through
  DOCUMENTURL:
    fstring: # built from a format string
      - "url_oa|https://doi.org/{doi}"

Paths such as attributes.subjects.*.0.text are glom expressions into the record, so nested and repeated values are reachable without code. Conditions for skip and withdrawal are built from all, any and not around leaves that test a field with equals, in, any_in or regex.

See docs/mapping-format.md for the full reference.

Output format

A root add element holding one doc per record, each a flat list of field elements carrying the value and the name of the field it belongs to:

<add>
  <doc>
    <field name="DATABASE">PIXLS</field>
    <field name="SUBDATABASE">biorxiv</field>
    <field name="TITLE">Toy example study on synthetic cell behavior</field>
    <field name="AUTHOR">Example, A.</field>
    <field name="AUTHOR">Sample, B.</field>
  </doc>
</add>

Withdrawn records travel in the same file as delete entries, which name only the record to remove:

<doc><entry><field name="id">PIXLS-biorxiv-10.1101/2025.07.15.664985</field></entry></doc>

Development

The Makefile wraps the common tasks:

make test                  # run the test suite
make reformat              # black, rewriting files
make check-formatting      # black, in check mode
make check-code            # flake8
make check-coverage        # pytest with coverage
make audit                 # formatting + lint + tests, the pre-merge gate

Project layout

livivo_converter/
├── cli.py          # argument parsing, logging, checks before work starts
├── mappings.py     # loading and schema validation of a mapping
├── convert.py      # executes a mapping over the input records
├── formatter.py    # resolves glom placeholders inside format strings
├── xml_builder.py  # turns mapped fields into the LIVIVO XML tree
└── mappings/       # one YAML mapping per source
    ├── biorxiv.yaml
    ├── medrxiv.yaml
    └── osf.yaml

Getting help

• Something broken, or a source converting wrongly? Open an issue at codeberg.org/pixls/livivo-converter/issues.
• Writing or changing a mapping: docs/mapping-format.md is the reference, and the three mappings that ship with the package are worked examples.
• A record that did not come out as expected: run with --verbose. Every skipped record is logged with the values its skip rule tested, and every converted one with the fields as they were built, which is usually enough to see which rule or path is at fault.

Contributing

Contributions are welcome. Please read CONTRIBUTING.md for the workflow, coding conventions and what make audit expects before you open a pull request.

Maintainers

Developed and maintained by Benjamin Wolff (wolff@zbmed.de) at ZB MED – Information Centre for Life Sciences, as part of the PIXLS project.

Funding

Developed in PIXLS – Preprint Information eXtraction for Life Sciences, funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation).

License

Source code is licensed under the MIT License. Documentation and other content are licensed under CC BY 4.0, and configuration and generated files under CC0 1.0. See LICENSING.md for details.