Skip to main content

microdata-tools

Tools for the microdata.no platform

microdata-tools is a Python library with two audiences:

  • Data owners preparing datasets for microdata.no: validate your data and metadata, then encrypt and package the files into a .tar archive ready for upload.
  • microdata.no services (such as job-executor) that receive packaged datasets: unpackage and decrypt them, and validate them on the receiving end.

A dataset normally contains:

  • A CSV file with one identifier, one value and the dates that describe when the value applies.
  • A JSON file describing the dataset, its variable and its values.

Installation

microdata-tools can be installed from PyPI using pip:

pip install microdata-tools

Documentation

If you're a data owner preparing a dataset, the full guide lives on the documentation site:

The sections below are aimed at developers integrating microdata-tools into another service (such as job-executor), and at contributors to this repository.

Unpackaging a dataset

The unpackage_dataset() function will reverse the packaging step: it untars the archive and use the combined ML-KEM-768/X25519 private key from microdata_private_key.pem to recover the symmetric key, which is then used to decrypt the data file.

The packaged file must have the <DATASET_NAME>.tar extension. Its contents should be:

  • <DATASET_NAME>.json: Required metadata file.
  • <DATASET_NAME>.csv.encr: Optional encrypted data file.
  • <DATASET_NAME>.kem.encr: Optional HPKE ciphertext file containing the encrypted symmetric key. Required if .csv.encr is present.

Decryption uses the combined ML-KEM-768/X25519 private key located at PRIVATE_KEY_DIR to recover the symmetric decryption key. The dataset is then stored in output_dir/archive/unpackaged after a successful run, or output_dir/archive/failed after an unsuccessful one. For details on how packaging encrypts a dataset in the first place, see the packaging format.

Contribute

Set up

To work on this repository you need to install uv:

# macOS / linux / BashOnWindows
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows powershell
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Then install the virtual environment from the root directory:

uv sync

Running unit tests

Open terminal and go to root directory of the project and run:

uv run pytest

Pre-commit

There are currently 3 active rules: Ruff-format, Ruff-lint and sync lock file. Install pre-commit

pip install pre-commit

If you've made changes to the pre-commit-config.yaml or its a new project install the hooks with:

pre-commit install

Now it should run when you do:

git commit

By default it only runs against changed files. To force the hooks to run against all files:

pre-commit run --all-files

if you dont have it installed on your system you can use: (but then it won't run when you use the git-cli)

uv run pre-commit

Read more about pre-commit

Metadata

Release files for microdata-tools 2.2.5

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for microdata-tools 2.2.5
File Size Uploaded Provenance Actions
microdata_tools-2.2.5.tar.gz 43.6 kB With provenance Details for microdata_tools-2.2.5.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for microdata-tools 2.2.5
File Interpreter ABI Platform Actions
microdata_tools-2.2.5-py3-none-any.whl Python 3 none any Details for microdata_tools-2.2.5-py3-none-any.whl

Total release size: 105.6 kB

Release files / microdata_tools-2.2.5.tar.gz

Download URL microdata_tools-2.2.5.tar.gz
Size 43.6 kB
Tags Source
SHA-256 checksum
How to use checksums
bbd3c34516b598d4507de1d9fb34b0748d905f6caa087aaf990b73ffc53f20a1
BLAKE2b-256 checksum
How to use checksums
ca6a9e1805cf03a0281cb7df49be74b871ecc730f729bc1130b073333be03129
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

Release files / microdata_tools-2.2.5-py3-none-any.whl

Download URL microdata_tools-2.2.5-py3-none-any.whl
Size 62.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
f4488494444fd7ca02c8a2cd9cd675017543d517ab3e889fd4f26e9ae5546978
BLAKE2b-256 checksum
How to use checksums
a78e8d36dd366483f73aa69d43d6b8d723d9d36346d239df765d0193379c00ae
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

Security for microdata-tools 2.2.5

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for microdata_tools-2.2.5.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 19, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

2.2.5 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page