sparv-sbx-wsd-rs

This plugin to sparv is a rewrite of the the internal module wsd, that use saldowsd-rs instead of saldowsd.jar.

The improvements to the internal module wsd are:

• Easier setup, the binary saldowsd is installed as a Python package from PyPI, instead of manual install of saldowsd.jar.
• Faster analysis, saldowsd-rs is about 13% faster than saldowsd.jar.
• Uses less memory, saldowsd-rs uses about 35% less memory than saldowsd.jar.

Faster than using saldowsd.jar

The running time for sbx_wsd_rs that uses saldowsd-rs is 12.8% faster than using Java version. See results from running both annotations on vivill on our server wombat.

[username@wombat vivill]$ sparv run --stats

  Task                                                           Time taken   Percentage
  wsd:annotate                                                      0:11:25         1.3%
  sbx_wsd_rs:annotate                                               0:09:57         1.1%

Memory usage

Loading models and running a simple example (not using Sparv for this). Rust version uses 35% less memory. Measured with heaptrack.

An example of the output from Sparv can be seen here.

The annotations are probalistic, so they always differ a bit (wsd.sense differs with itself for different runs).

Example of differences:

• anslag: |anslag..1:0.993|anslag..2:0.004|anslag..3:0.004| != |anslag..1:0.992|anslag..3:0.004|anslag..2:0.004|
• avvikelse: |avvikelse..1:0.978|avvikelse..2:0.022| != |avvikelse..1:0.977|avvikelse..2:0.023|
• utskottets: |utskott..2:0.835|utskott..3:0.109|utskott..1:0.056| != |utskott..2:0.843|utskott..3:0.101|utskott..1:0.056|
• särskilda: |särskilja..1:0.587|särskild..1:0.413| != |särskilja..1:0.589|särskild..1:0.411|

Changelog

This project keeps a changelog.

Minimum supported Python version

This library tries to support as many Python versions as possible. When a Python version is added or dropped, this library's minor version is bumped.

• v0.1.0: Python 3.11

License

This repository is licensed under the MIT license.

Development

Development prerequisites

• uv
• pre-commit

For starting to develop on this repository:

• Clone the repo git clone https://github.com/spraakbanken/sparv-sbx-wsd-rs.git
• Setup environment: make dev
• Install pre-commit hooks: pre-commit install

Do your work.

Tasks to do:

• Test the code with make test or make test-w-coverage.
• Test the examples with make test-example-small.
• Lint the code with make lint.
• Check formatting with make check-fmt.
• Format the code with make fmt.
• Type-check the code with make type-check.

This repo uses conventional commits.

Release a new version

• Prepare the CHANGELOG: make prepare-release and then edit CHANGELOG.md.
• Add to git: git add CHANGELOG.md
• Commit with git commit -m 'chore(release): prepare release' or cog commit chore 'prepare release' release.
• Bump version (depends on `bump-my-version)
  • install with uv tool install bump-my-version
  • Major: make bumpversion part=major
  • Minor: make bumpversion part=minor
  • Patch: make bumpversion part=patch or make bumpversion
• Push main and tags to GitHub: git push main --tags or make publish
  • GitHub Actions will build, test and publish the package to PyPi.
• Add metadata for Språkbanken's resource
  • Generate metadata: make generate-metadata
  • Upload the files from examples/metadata/export/sbx_metadata/analysis to https://github.com/spraakbanken/metadata/tree/main/yaml/analysis.