sws-spark-dissemination-helper
Python/PySpark helper library for the FAO SWS (Statistical Working System) data dissemination pipelines. It provides a set of classes that move a dataset from the SWS Postgres database, through an Iceberg-backed Bronze → Silver → Gold medallion pipeline, to the CSV/Iceberg artifacts and SWS "dissemination tags" consumed downstream (including SDMX and FAOSTAT-shaped outputs).
Pipeline overview
| Layer | Class | Role |
|---|---|---|
| Source | SWSPostgresSparkReader |
Reads SWS Postgres tables (observations, codelists, datatables) into Spark and stages them as Iceberg tables. Every stage reads the specific datatables it needs on demand through this class, rather than a single upfront import. |
| Bronze | SWSBronzeIcebergSparkHelper |
Denormalizes the raw observation/coordinate/metadata tables into one wide table (codes instead of ids, metadata and unit-of-measure attached). Can also be filtered to a dimension-value selection provided as input and republished as a separate "disseminated tag" table. |
| Silver | SWSSilverIcebergSparkHelper |
Applies dissemination business rules on top of Bronze data: code correction, time-validity checks, dissemination flags/exceptions, duplicate detection. |
| Gold | SWSGoldIcebergSparkHelper |
Produces the published output shapes from Silver/Bronze data: SWS variants, a pre-SDMX/FMR-layout variant (SDMX doesn't accept dots in coded values and requires the value column named OBS_VALUE), an SDMX-shaped variant, and FAOSTAT variants — the SDMX and FAOSTAT variants are both produced by mapping the pre-SDMX output through a remote SDMX structure-map chain run as SWS plugin tasks. Also applies display-decimal rounding. |
| Standalone | SWSEasyIcebergSparkHelper |
Exports a huge dataset to Iceberg as a single denormalized table, for datasets large enough that querying them directly in SWS is impractical or impossible. |
Each helper writes its output to Iceberg, creates a version tag, optionally caches a CSV copy on S3, and can register the result in the SWS dissemination-tag metadata via the sws_api_client Tags API. See each class's docstring for the exact tables/columns it reads and writes.
Installation
pip install sws-spark-dissemination-helper
Requires Python 3.9 or 3.11 (see pyproject.toml).
Construct SWSPostgresSparkReader once per job, then pass it into whichever helper class(es) you need for the pipeline stage(s) you're running — see the architecture page for how the stages relate and the API reference (or each class's docstring) for the exact constructor arguments.
Configuration
The package expects to run on Amazon EMR, with AWS access already configured (an EMR instance profile):
EMR_BUCKET: S3 bucket used as the Spark/Iceberg warehouse location (seeutils.get_spark).DB_SECRET: name of the AWS Secrets Manager secret holding the Postgres JDBC connection properties (host,port,database,user,password), used whenSWSPostgresSparkReaderis instantiated without explicitjdbc_conn_properties.- AWS credentials are picked up from the default
boto3credential chain and used both for S3/Glue access and to build the temporary Sparks3acredentials.
Documentation
The full documentation (architecture, configuration, and the generated API reference) is built with Sphinx from this README and the package's docstrings. Build it locally with:
pip install -e ".[docs]"
make docs
Then open docs/build/html/index.html. See docs/source/ for the source.
How to use (development)
Make sure to have make installed on your system and run make help to see the available commands.
Upload a new version
A new version is published to PyPI automatically by the publish workflow whenever a GitHub release is published. The workflow verifies that the release tag matches v<project.version>, so the two must always be kept in sync.
-
Update
project.versionin the pyproject.toml file (e.g.0.1.9.dev0) and commit the change. -
Create and push a tag matching the new version (note the leading
v):git tag v0.1.9.dev0 git push origin --tags
-
Go to the tags page, select the tag and create a release from it. Use the Latest release label for official releases or Pre-release for development versions, then fill in the release notes.
-
Wait for the GitHub action to run. If it succeeds the new version will be available on PyPI; otherwise check the workflow logs for the failure reason (a common one is the tag not matching
project.version) and fix it.
2 maintainers
Metadata
2 maintainers
Release files for sws-spark-dissemination-helper 0.1.18
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| sws_spark_dissemination_helper-0.1.18.tar.gz | 87.0 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| sws_spark_dissemination_helper-0.1.18-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 163.1 kB
Release files / sws_spark_dissemination_helper-0.1.18.tar.gz
| Download URL | sws_spark_dissemination_helper-0.1.18.tar.gz |
|---|---|
| Size | 87.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3264cfc183129b2505390c1f3becf2a425910da146ff807d1ae5a8b1d68fe52a
|
|
BLAKE2b-256 checksum How to use checksums |
1b3d43b2d0833f1e7396ab0969939ede9c90a02cff057014ea0a5c22fa1f22b1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 23, 2026.
Transparency logRelease files / sws_spark_dissemination_helper-0.1.18-py3-none-any.whl
| Download URL | sws_spark_dissemination_helper-0.1.18-py3-none-any.whl |
|---|---|
| Size | 76.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
4bf7791e91bae095d5225da4b9ffc0fc6f626a57526c58d494d25c7cb7bbafa2
|
|
BLAKE2b-256 checksum How to use checksums |
984493d7d4530cb23414887da065ee34f4c0549b723d5f19181343239bd47e15
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Sep 23, 2026.
Transparency log
