microdata-tools
Tools for the microdata.no platform
microdata-tools is a Python library with two audiences:
- Data owners preparing datasets for microdata.no: validate your data and metadata, then encrypt and package the files into a
.tararchive ready for upload. - microdata.no services (such as job-executor) that receive packaged datasets: unpackage and decrypt them, and validate them on the receiving end.
A dataset normally contains:
- A CSV file with one identifier, one value and the dates that describe when the value applies.
- A JSON file describing the dataset, its variable and its values.
Installation
microdata-tools can be installed from PyPI using pip:
pip install microdata-tools
Documentation
If you're a data owner preparing a dataset, the full guide lives on the documentation site:
- Getting started — install, prepare, validate and package your dataset, step by step.
- The metadata model — a field-by-field reference for the metadata JSON file.
- The data file format — a reference for the CSV data file, including validation rules.
- The packaging format — how your data is encrypted and packaged.
The sections below are aimed at developers integrating microdata-tools into another service (such as job-executor), and at contributors to this repository.
Unpackaging a dataset
The unpackage_dataset() function will reverse the packaging step: it untars the archive and use the combined ML-KEM-768/X25519 private key from microdata_private_key.pem to recover the symmetric key, which is then used to decrypt the data file.
The packaged file must have the <DATASET_NAME>.tar extension. Its contents should be:
<DATASET_NAME>.json: Required metadata file.<DATASET_NAME>.csv.encr: Optional encrypted data file.<DATASET_NAME>.kem.encr: Optional HPKE ciphertext file containing the encrypted symmetric key. Required if.csv.encris present.
Decryption uses the combined ML-KEM-768/X25519 private key located at PRIVATE_KEY_DIR to recover the symmetric decryption key. The dataset is then stored in output_dir/archive/unpackaged after a successful run, or output_dir/archive/failed after an unsuccessful one.
For details on how packaging encrypts a dataset in the first place, see the packaging format.
Contribute
Set up
To work on this repository you need to install uv:
# macOS / linux / BashOnWindows
curl -LsSf https://astral.sh/uv/install.sh | sh
# Windows powershell
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"
Then install the virtual environment from the root directory:
uv sync
Running unit tests
Open terminal and go to root directory of the project and run:
uv run pytest
Pre-commit
There are currently 3 active rules: Ruff-format, Ruff-lint and sync lock file. Install pre-commit
pip install pre-commit
If you've made changes to the pre-commit-config.yaml or its a new project install the hooks with:
pre-commit install
Now it should run when you do:
git commit
By default it only runs against changed files. To force the hooks to run against all files:
pre-commit run --all-files
if you dont have it installed on your system you can use: (but then it won't run when you use the git-cli)
uv run pre-commit
Read more about pre-commit
Metadata
Release files for microdata-tools 2.2.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| microdata_tools-2.2.5.tar.gz | 43.6 kB | With provenance | Details for microdata_tools-2.2.5.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| microdata_tools-2.2.5-py3-none-any.whl | Python 3 | none | any | Details for microdata_tools-2.2.5-py3-none-any.whl |
Total release size: 105.6 kB
Release files / microdata_tools-2.2.5.tar.gz
| Download URL | microdata_tools-2.2.5.tar.gz |
|---|---|
| Size | 43.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
bbd3c34516b598d4507de1d9fb34b0748d905f6caa087aaf990b73ffc53f20a1
|
|
BLAKE2b-256 checksum How to use checksums |
ca6a9e1805cf03a0281cb7df49be74b871ecc730f729bc1130b073333be03129
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.
Transparency logRelease files / microdata_tools-2.2.5-py3-none-any.whl
| Download URL | microdata_tools-2.2.5-py3-none-any.whl |
|---|---|
| Size | 62.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f4488494444fd7ca02c8a2cd9cd675017543d517ab3e889fd4f26e9ae5546978
|
|
BLAKE2b-256 checksum How to use checksums |
a78e8d36dd366483f73aa69d43d6b8d723d9d36346d239df765d0193379c00ae
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.
Transparency logSecurity for microdata-tools 2.2.5
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for microdata_tools-2.2.5.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.