sparv-sbx-conllu 0.2.0

Add your description here

sparv-sbx-conllu

Plugin to sparv import CoNLL-U files to Sparv.

Install

First, install sparv as suggested,

pipx install sparv

Then install sparv-sbx-conllu with

sparv plugins install sparv-sbx-conllu

Usage

To use this plugin to import CoNLL-U files to your corpus, add the following to your config.yaml:

# file=config.yaml
import:
  importer: sbx_conllu:parse

Configuration

All annotations are exported by default, but if you want to use a annotation in another analysis you need to specify them in your config.yaml.

By default, sparv_sbx_conllu exports the annotations text, document, sentenceand token. Other annotations can be added to sbx_conllu.import_attributes.

For instance, to use the annotations token:xpos and sentence:sent_id add them like this:

# file=config.yaml
sbx_conllu:
  import_attributes:
    - token:xpos
    - sentence:sent_id

Classes

To use annotations from sparv_sbx_conllu in other analysis you can be needed to add them to classes in your config.yaml.

Example

You want to use the sentence and token from sparv_sbx_conllu and also map token:xpos as token:pos and token:pos_ud as token:upos.

# file=config.yaml
classes:
  sentence: sentence
  token: token
  "token:pos": token:xpos
  "token:upos": token:pos_ud

Common annotation from a standard CoNLL-U file

These annotations are some of the annotations you can expect from a CoNLL-U file, but it depends on the file.

| Annotation | CoNLL-U column | Always? | Comment | Example in CoNLL-U | | ------------------------------------- | ------------------------------------------- | ------- | ------------------------------------------------------ | -------------------------------------------- | ---------- | ----------- | ------------- | | text | | yes | the whole text | | document | | yes | either implicit, or at least one specified | | sentence | | yes | must contain at least one sentence | | token | | yes | must contain at least one token | | document:id | # newdoc id = | no | | # newdoc id = ID gives document:id = ID | | paragraph | # newpar
NewPar=Yes in column misc | no | Can exist around sentences
And inside sentences | # newpar
NewPar=Yes in misc column | | paragraph:id | # newpar id = | no | | # newpar id = ID gives paragraph:id = ID | | token:id | id column | no | Always present in standard CoNLL-U | | token span over this form in the text | form column | no | Always present in standard CoNLL-U, may contain spaces | | token:baseform | lemma column | no | May contain spaces | | token:pos_ud | upos column | no | UD POS | | token:xpos | xpos column | no | custom POS (no standard) | | token:feats_ud | feats column | no | Dict-like values | Case=Nom | Gender=Fem | Number=Sing | Polarity=Pos | | token:dephead_ud | head column | no | integer | | token:deprel_ud | deprel column | no | UD-dep value | | token:deps_ud | depscolumn | no | At least one pairhead:deprel |2:obj | 4:obj | |token:misc_ud |misccolumn | no | Dict-like values |SpaceAfter=No` |

Known issues

Importing CoNLL-U

Sparv will log a warning if any of the following are encountered.

• When importing CoNLL-U data, empty nodes are skipped. E.g. id = 5.1 Tracking issue
• When importing CoNLL-U data, tokens inside multiword tokens are merged, see Multiword. E.g. id = 2-3 are added, but id = 2 and id = 3 are skipped. Tracking issue

Minimum Supported Python Version Policy

The Minimum Supported Python Version is fixed for a given minor (1.x) version. However it can be increased when bumping minor versions, i.e. going from 1.0 to 1.1 allows us to increase the Minimum Supported Python Version. Users unable to increase their Python version can use an older minor version instead. Below is a list of sparv-sbx-conllu versions and their Minimum Supported Python Version:

• v0.1: Python 3.11.

Note however that sparv-sbx-conllu also has dependencies, which might have different MSPV policies. We try to stick to the above policy when updating dependencies, but this is not always possible.

Changelog

This project keeps a changelog.

License

This repository is licensed under the MIT license.

Development

Development prerequisites

• uv
• pre-commit

For starting to develop on this repository:

• Clone the repo (in one of the ways below):
  • git clone [e-mail]:spraakbanken/sparv-sbx-conllu.git
  • git clone https://github.com/spraakbanken/sparv-sbx-conllu.git
• Setup environment: make dev
• Install pre-commit hooks: pre-commit install

Do your work.

Tasks to do:

• Test the code with make test or make test-w-coverage.
  • Snapshot can be updated by make snapshot-update
• Lint the code with make lint.
• Check formatting with make check-fmt.
• Format the code with make fmt.
• Type-check the code with make type-check.
• Test the examples with:
  • make test-example-en_ewt-ud-test
  • make test-example-long-token-to-text
  • make test-example-no-metadata
  • make test-example-paragraph-and-document
  • make test-example-paragraph-in-sentence
  • make test-example-sentence-comments

This repo uses conventional commits.

Release a new version

• Prepare the CHANGELOG: make prepare-release.
• Edit CHANGELOG.md to your liking.
• Add to git: git add --update
• Commit with git commit -m 'chore(release): prepare release' or cog commit chore 'prepare release' release.
• Bump version (depends on `bump-my-version)
  • install with uv tool install bump-my-version
  • Major: make bumpversion part=major
  • Minor: make bumpversion part=minor
  • Patch: make bumpversion part=patch or make bumpversion
• Push main and tags to GitHub: git push main --tags or make publish
  • GitHub Actions workflow will build, test and publish the package to PyPi.
• Add metadata for Språkbanken's resource
  • Generate metadata: make generate-metadata
  • Upload the files from assets/metadata/export/sbx_metadata/utility to https://github.com/spraakbanken/metadata/tree/main/yaml/utility.