microdata-tools

Tools for the microdata.no platform

Installation

microdata-tools can be installed from PyPI using pip:

pip install microdata-tools

Usage

Once you have your metadata and data files ready to go, they should be named and stored like this:

my-input-directory/
    MY_DATASET_NAME/
        MY_DATASET_NAME.csv
        MY_DATASET_NAME.json

The CSV file is optional in some cases.

Package dataset

The package_dataset() function will encrypt and package your dataset as a tar archive. The process is as follows:

  1. Generate the symmetric key for a dataset.
  2. Encrypt the dataset data (CSV) using an AES-256-GCM symmetric key and store the encrypted file as <DATASET_NAME>.csv.encr
  3. Encrypt the symmetric key using HPKE with the combined ML-KEM-768/X25519 public key from microdata_public_key.pem and store the resulting HPKE ciphertext as <DATASET_NAME>.kem.encr
  4. Gather the encrypted CSV, ciphertext file and metadata (JSON) file in one tar file.

Unpackage dataset

The unpackage_dataset() function will untar and your dataset and use the combined ML-KEM-768/X25519 private key from microdata_private_key.pem to recover the symmetric key, which is then used to decrypt the dataset.

The packaged file has to have the <DATASET_NAME>.tar extension. Its contents should be as follows:

<DATASET_NAME>.json : Required medata file.

<DATASET_NAME>.csv.encr : Optional encrypted dataset file.

<DATASET_NAME>.kem.encr : Optional HPKE ciphertext file containing the encrypted symmetric key required to decrypt the dataset file. Required if the .csv.encr file is present.

Decryption uses the combined ML-KEM-768/X25519 private key located at PRIVATE_KEY_DIR to recover the symmetric decryption key.

The packaged file is then stored in output_dir/archive/unpackaged after a successful run or output_dir/archive/failed after an unsuccessful run.

Example

Store your metadata and data files according to the structure described above, and put the provided public key in a directory of your choice. Then:

from pathlib import Path
from microdata_tools import package_dataset

package_dataset(
    public_key_dir=Path("path/to/key_directory"),
    dataset_dir=Path("path/to/MY_DATASET_NAME"),
    output_dir=Path("path/to/output"),
)

This produces path/to/output/MY_DATASET_NAME.tar, which can be uploaded to microdata.

Validation

Once you have your metadata and data files ready to go, they should be named and stored like this:

my-input-directory/
    MY_DATASET_NAME/
        MY_DATASET_NAME.csv
        MY_DATASET_NAME.json

Note that the filename only allows upper case letters A-Z, number 0-9 and underscores.

Import microdata-tools in your script and validate your files:

from microdata_tools import validate_dataset

validation_errors = validate_dataset(
    "MY_DATASET_NAME",
    input_directory="path/to/my-input-directory"
)

if not validation_errors:
    print("My dataset is valid")
else:
    print("Dataset is invalid :(")
    # You can print your errors like this:
    for error in validation_errors:
        print(error)

For a more in-depth explanation of usage visit the usage documentation.

Data format description

A dataset as defined in microdata consists of one data file, and one metadata file.

The data file is a csv file seperated by semicolons. A valid example would be:

000000000000001;123;2020-01-01;2020-12-31;
000000000000002;123;2020-01-01;2020-12-31;
000000000000003;123;2020-01-01;2020-12-31;
000000000000004;123;2020-01-01;2020-12-31;

Read more about the data format and columns in the documentation.

The metadata files should be in json format. The requirements for the metadata is best described through the Pydantic model, the examples, and the metadata model.

Contribute

Set up

To work on this repository you need to install uv:

# macOS / linux / BashOnWindows
curl -LsSf https://astral.sh/uv/install.sh | sh

# Windows powershell
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

Then install the virtual environment from the root directory:

uv sync

Running unit tests

Open terminal and go to root directory of the project and run:

uv run pytest

Pre-commit

There are currently 3 active rules: Ruff-format, Ruff-lint and sync lock file. Install pre-commit

pip install pre-commit

If you've made changes to the pre-commit-config.yaml or its a new project install the hooks with:

pre-commit install

Now it should run when you do:

git commit

By default it only runs against changed files. To force the hooks to run against all files:

pre-commit run --all-files

if you dont have it installed on your system you can use: (but then it won't run when you use the git-cli)

uv run pre-commit

Read more about pre-commit

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

Owner

PyPI data
Data sourced directly from PyPI's database.

Credits

Author: microdata-developers

License

MIT

Requires

Python >=3.10

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

microdata_tools-2.0.1.tar.gz (43.1 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

microdata_tools-2.0.1-py3-none-any.whl (61.3 kB view details)

Uploaded Python 3

File details

Details for the file microdata_tools-2.0.1.tar.gz.

File metadata

  • Download URL: microdata_tools-2.0.1.tar.gz
  • Upload date:
  • Size: 43.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for microdata_tools-2.0.1.tar.gz
Algorithm Hash digest
SHA256 f8957c6915f8eee9bcecc789482a9b2460908a492b6d9f69fee19e752336bf3a
MD5 bd8b494f0d0da0637b1aae8a153e9d56
BLAKE2b-256 1e0bca36a75fbbe5521777ef39669016b07e56ed99e297032b7ec6a1d3a5bab2

See more details on using hashes here.

Provenance

The following attestation bundles were made for microdata_tools-2.0.1.tar.gz:

Publisher: test-and-publish.yaml on statisticsnorway/microdata-tools

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file microdata_tools-2.0.1-py3-none-any.whl.

File metadata

  • Download URL: microdata_tools-2.0.1-py3-none-any.whl
  • Upload date:
  • Size: 61.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.14

File hashes

Hashes for microdata_tools-2.0.1-py3-none-any.whl
Algorithm Hash digest
SHA256 38649271067e35f9b6511f6b9d32bf93f47b50cd2dba5a1a94ab033f44005501
MD5 2b74ab3c50d261778d2bbbc13215c78b
BLAKE2b-256 16f574b85506b7640d041454c22f4c7f42983a14d2243d075ae42c309b651daa

See more details on using hashes here.

Provenance

The following attestation bundles were made for microdata_tools-2.0.1-py3-none-any.whl:

Publisher: test-and-publish.yaml on statisticsnorway/microdata-tools

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

This release

2.0.1 This release

2 files

2.0.0

2 files

1.14.0

2 files

1.13.1

2 files

1.13.0

2 files

1.12.1

2 files

1.12.0

2 files

1.11.0

2 files

1.10.8

2 files

1.10.7

2 files

1.10.6

2 files

1.10.5

2 files

1.10.4

2 files

1.10.3

2 files

1.10.2

2 files

1.10.1

2 files

1.10.0

2 files

1.9.0

2 files

1.8.0

2 files

1.7.1

2 files

1.7.0

2 files

1.6.0

2 files

1.5.0

2 files

1.4.0

2 files

1.3.0

2 files

1.2.0

2 files

1.1.1

2 files

1.1.0

2 files

1.0.4

2 files

1.0.3

2 files

1.0.2

2 files

1.0.1

2 files

1.0.0

2 files

0.18.0

2 files

0.17.0

2 files

0.16.0

2 files

0.15.0

2 files

0.14.1

2 files

0.14.0

2 files

0.13.3

2 files

0.13.2

2 files

0.13.1

2 files

0.13.0

2 files

0.12.1

2 files

0.12.0

2 files

0.11.7

2 files

0.11.6

2 files

0.11.5

2 files

0.11.4

2 files

0.11.3

2 files

0.11.2

2 files

0.11.1

2 files

0.11.0

2 files

0.10.0

2 files

0.9.1

2 files

0.9.0

2 files

0.8.2

2 files

0.8.1

2 files

0.8.0

2 files

0.7.3

2 files

0.7.2

2 files

0.7.1

2 files

0.7.0

2 files

0.6.0

2 files

0.5.1

2 files

0.5.0

2 files

0.4.0

2 files

0.3.0

2 files

0.2.0

2 files

0.1.0

2 files

0.0.1

2 files

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page