Skip to main content

dcp-tools

Manage and load data to Data Commons Platform instances

PyPI PyPI - Python Version Docs Lint/format: Ruff

A Data Commons Platform instance takes your data as CSVs in a fixed variable-per-row shape, plus a config.json that maps each CSV's columns onto the Data Commons schema. If you're defining your own statistical variables, entities, or properties rather than reusing existing ones, you also need MCF (Meta Content Framework) files describing them. dcp-tools builds and validates that bundle in Python (or from the CLI) and uploads it to Cloud Storage to trigger the platform's ingestion workflow.

This package was published as bblocks-datacommons-tools until version 0.1.1, and imported as bblocks.datacommons_tools. Installing the old distribution now pulls in dcp-tools and redirects those imports with a DeprecationWarning, so existing code keeps working. Update imports to dcp_tools when convenient.

Install

pip install dcp-tools

Or from GitHub:

pip install git+https://github.com/ONEcampaign/dcp-tools

Quickstart

This builds a config and MCF for a source, a provenance, one input file, and one statistical variable, then exports the bundle to disk.

from pathlib import Path

import pandas as pd
from dcp_tools import CustomDataManager

manager = CustomDataManager()
manager.add_source(dcid="ONEData", url="https://data.one.org")
manager.add_provenance(
    dcid="ONEClimateFinance",
    url="https://datacommons.one.org/data/climate-finance-files",
    source="ONEData",
)

data = pd.DataFrame({
    "country": ["Kenya", "Kenya", "Vietnam"],
    "year": [2022, 2023, 2023],
    "variable": ["climateFinanceProvidedCommitments"] * 3,
    "value": [12.4, 15.1, 8.7],
})
manager.add_input_file(
    file_name="climate_finance/one_cf_provider_commitments.csv",
    provenance="ONEClimateFinance",
    data=data,
    column_mappings={
        "observationAbout": "country",
        "date": "year",
        "variable": "variable",
        "value": "value",
    },
    observation_properties={"unit": "USDollar"},
)

manager.add_variable_to_mcf(
    dcid="climateFinanceProvidedCommitments",
    name="Climate finance commitments (bilateral)",
    description="Funding committed for climate adaptation and mitigation projects",
    stat_type="dcid:measuredValue",
)

manager.add_mcf_file("*.mcf", provenance="ONEClimateFinance")

out_dir = Path("export/climate_finance")
out_dir.mkdir(parents=True, exist_ok=True)
manager.export_all(out_dir)

export_all writes config.json, the CSV, and both MCF files under out_dir. Since we never called set_import_name, config.json defaults importName to the export directory's name, and column mappings and the provenance name are resolved to full dcids:

{
    "importName": "climate_finance",
    "inputFiles": [
        {
            "filename": "climate_finance/one_cf_provider_commitments.csv",
            "provenance": "dcid:provenance/ONEClimateFinance",
            "columnMappings": {
                "dcid:variableMeasured": "variable",
                "dcid:observationDate": "year",
                "dcid:value": "value",
                "dcid:observationAbout": "country"
            },
            "observationProperties": {"unit": "USDollar"},
            "format": "variablePerRow"
        },
        {
            "pattern": "*.mcf",
            "provenance": "dcid:provenance/ONEClimateFinance"
        }
    ]
}

Loading it

Once you have a bundle on disk, dcp_tools.gcp_utilities uploads it and triggers ingestion:

from dcp_tools.gcp_utilities import (
    get_kg_settings,
    run_ingestion_workflow,
    upload_to_cloud_storage,
)

settings = get_kg_settings(source="env", env_file="customDC.env")
upload_to_cloud_storage(settings=settings, directory="export/climate_finance")
run_ingestion_workflow(settings=settings)

run_ingestion_workflow triggers the DCP (Data Commons Platform) ingestion workflow, which ingests the new data and serves it. dcp-tools 1.0.x pairs with DCP stacks up to 1.1.5, and dcp-tools 1.1.x with DCP 1.1.6. Starting an ingestion uses Application Default Credentials, or impersonates ingestion_service_account, and ignores gcp_credentials. run_ingestion_workflow returns the execution name as soon as the run starts. Pass it to wait_for_ingestion to block until the run finishes. See the loading-data docs for the full settings reference, and the dcp-tools CLI (upload, ingest, pipeline), which wraps this same flow.

Contributing

Contributions are welcome! See CONTRIBUTING for how to get started, report bugs, and submit changes.

Metadata

Release files for dcp-tools 1.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for dcp-tools 1.2.0
File Size Uploaded Provenance Actions
dcp_tools-1.2.0.tar.gz 52.5 kB Without provenance Details for dcp_tools-1.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for dcp-tools 1.2.0
File Interpreter ABI Platform Actions
dcp_tools-1.2.0-py3-none-any.whl Python 3 none any Details for dcp_tools-1.2.0-py3-none-any.whl

Total release size: 115.8 kB

Release files / dcp_tools-1.2.0.tar.gz

Download URL dcp_tools-1.2.0.tar.gz
Size 52.5 kB
Tags Source
SHA-256 checksum
How to use checksums
aa33c778073702e1a13c426f4b4697eaa78db88ee8f6a657601e31ce04d17ab0
BLAKE2b-256 checksum
How to use checksums
4d9ec590ff9ca6ef71e1dcd57f1d2b89a07bde18f10b03077e68fcf5d0e9b3db
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.5 {"installer":{"name":"uv","version":"0.13.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / dcp_tools-1.2.0-py3-none-any.whl

Download URL dcp_tools-1.2.0-py3-none-any.whl
Size 63.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
b9e4d1e96d02422bb8a9cdb93e5e339282309be316a3d53ee91e7c2dfdf9d6ac
BLAKE2b-256 checksum
How to use checksums
2316aaf8d97a3398fcf9d32930642e1e00f8f2ff9c801cf60e8a6b26817e35d8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.5 {"installer":{"name":"uv","version":"0.13.5","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for dcp-tools 1.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page