cf-datahive 0.1.11

Canonical result and measurement data storage APIs for Cogniflow

cf_datahive

cf_datahive owns the Data Hive write provider for CfDataHiveSinkContractV1, publishes the read-side datahive_runs service contract for first-party consumers, and exposes a local Python read/inspect API for manual inspection, owner-side tooling, and package-local validation.

For first-party package integration the boundary is strict:

• cross-package writes go only through cf_package_contracts and injected contract slots such as CfExecutionContext.datahive_sink
• cross-package reads go only through cf_service_contracts and the datahive_runs provider key
• first-party packages must not import cf_datahive
• first-party packages must not reach into cf_datahive/native/* or consume cf_datahive_cpp_* helper APIs

Boundary

• sandcastle/cf_datahive/src/cf_datahive/cpp is the owner-side native gatekeeper for writes under workspace/data_hive
• sandcastle/cf_datahive/src/cf_datahive/service_contract_provider.py publishes the canonical read-side datahive_runs contract for first-party consumers
• sandcastle/cf_datahive/src/cf_datahive provides owner-side read/inspect APIs such as DataHiveClient
• CfDataHiveSinkContractV1 in cf_package_contracts is the only supported cross-package sink interface
• DataHiveRunsContractV1 in cf_service_contracts is the only supported cross-package read interface

First-party packages that need read-side runtime integration should resolve ProviderKey.DATAHIVE_RUNS against the canonical workspace root instead of importing cf_datahive directly.

Usage

First-party read-side integration:

from cf_service_contracts import ProviderKey, require_provider

datahive_runs = require_provider(ProviderKey.DATAHIVE_RUNS)
evidence = datahive_runs.find_latest_run(workspace_root, pipeline_id)
if evidence is not None:
  print(evidence["run_id"], evidence["manifest_status"])

Owner-side inspection workflow:

from pathlib import Path

from cf_datahive import DataHiveClient

workspace_root = Path("workspace")
client = DataHiveClient(str(workspace_root))

runs = client.list_runs("opcua_fifo_avg")
if runs:
    latest = runs[0].run_id
    manifest = client.load_manifest("opcua_fifo_avg", latest)
    table = client.read_table("opcua_fifo_avg", latest, "measurements")
    print(manifest.status, table.num_rows)

DataHiveClient, RunHandle, and StorageModePolicy remain public for owner-side workflows, but they are not a first-party integration surface.

Canonical Layout

workspace/
  data_hive/
    <pipeline_id>/
      <yyyy>/<mm>/<dd>/<session_id>/
        runs/<run_id>/manifest.json
        runs/<run_id>/artifacts/<artifact_name>
        tables/<table_name>/run_id=<run_id>/part-0000.parquet
        tables/<table_name>/_session_compacted/part-0000.parquet
        session_manifest.json
      latest.txt
      latest_session.txt

• latest.txt stores the latest committed run_id
• latest_session.txt stores the latest committed session_id
• manifest.json is the run SOT for tables, artifacts, hashes, and session linkage
• session_manifest.json is the per-session SOT for storage-mode locks
• flat run layouts and direct data_hive/ roots outside the canonical workspace root are not supported

Local Build Notes

cf_datahive now consumes the workspace-store runtime contract instead of linking the storage backend directly. Editable builds still use the shared native dependency cache under ~/.cogniflow/cache/native/ and the standard cf_setup MSVC/CMake environment preparation.

Build caches are expected under ~/.cogniflow/cache/native/scikit-build/cf-datahive.

Guardrails

Run:

python tools/check_datahive_guardrails.py

The guardrail script hard-fails when first-party code outside sandcastle/cf_datahive:

• imports cf_datahive
• references removed cf_datahive_cpp_* helper APIs
• reaches into the removed native consumer packaging surface

Testing

pip install -e "sandcastle/cf_datahive[test]"
pytest -q sandcastle/cf_datahive/tests

Published distribution name:

pip install cf-datahive

Publishing

cf_datahive still publishes the provider binary plus provider descriptor used by cf_setup and cf_pipeline_engine.

• Workflow: .github/workflows/cf_datahive_windows_publish.yml
• Package directory: sandcastle/cf_datahive
• PyPI tag: cf-datahive-v<version>
• TestPyPI tag: cf-datahive-v<version>-test