bCLEARer™ Pipeline Development Kit (PDK)

The bCLEARer Pipeline Development Kit (PDK) bundles the libraries, scaffolding tools, and reference assets used to build semantic data pipelines on the bCLEARer platform. It delivers the core building blocks for configuration, data interoperability, orchestration, and ontology modelling so you can go from a pipeline blueprint to a running implementation quickly.

Highlights

• Generate complete pipeline skeletons with the bclearer-pipeline-builder CLI (interactive authoring, JSON-driven creation, structural updates, and template extraction).
• Connect to the ecosystems your pipelines touch: CSV/Excel/JSON, Delta Lake, PySpark, HDF5, MongoDB, MS Access, PostgreSQL, SQL Server, Neo4j, CozoDB, Raphtory, Enterprise Architect, and more.
• Operate pipelines confidently with orchestration helpers covering app lifecycle management, UUID/identity services, logging, reporting, static analysis, version-control utilities, and unit-of-measure management.
• Model your universe with the BNOP ontology module, our BORO™ Native Objects implementation featuring factories, relationship management, and XML migrations.

Workspace packages

Repository layout

• libraries/ – the four workspace packages listed above.
• documentation/ – the pipeline framework, the clean coding and testing standards, and the interop manual.
• LICENSES/ – the full text of the AGPL-3.0-or-later licence.
• release_management/ – the PyPI publishing script and the distribution secret scanner.

Getting started

Requires Python 3.12+.

WSL prerequisites

If running on WSL, install the ODBC driver library before installing Python dependencies:

sudo apt-get update && sudo apt-get install -y unixodbc-dev

Without this, pyodbc will fail with ImportError: libodbc.so.2: cannot open shared object file: No such file or directory.

Install the workspace (recommended)

pip install uv
uv sync
source .venv/bin/activate

uv sync installs all workspace members (bclearer-core, bclearer-interop-services, bclearer-orchestration-services, bnop) in editable mode.

Alternative: standard pip

python -m venv .venv
source .venv/bin/activate
pip install -e .

Install individual packages from PyPI if you only need a subset, for example pip install bclearer-core.

Pipeline builder CLI

The pipeline builder turns JSON (or interactive prompts) into a fully structured bCLEARer pipeline: domains, pipelines, thin slices, stages, sub-stages, orchestrators, and b-units.

# Generate a sample configuration file
bclearer-pipeline-builder sample --output pipeline_config.json

# Create a pipeline in the current directory
bclearer-pipeline-builder create --config pipeline_config.json --output ./pipelines

# Update an existing pipeline from configuration
bclearer-pipeline-builder update --config pipeline_config.json --pipeline ./pipelines/example_domain

# Extract templates from a curated pipeline
bclearer-pipeline-builder update-templates --template-path pipelines/template_pipeline

Run bclearer-pipeline-builder help or python -m bclearer_core.pipeline_builder help for the full command reference. The generated pipelines follow the bCLEARer pipeline framework.

Working with the libraries

Data interchange

from bclearer_interop_services.b_dictionary_service.table_as_dictionary_service import (
    TableAsDictionaryFromCsvFileReader,
    TableAsDictionaryToDataFrameConverter,
)

reader = TableAsDictionaryFromCsvFileReader()
table_dict = reader.read("data/example.csv")

converter = TableAsDictionaryToDataFrameConverter()
dataframe = converter.convert(table_dict)

Beyond CSV and DataFrames you will find adapters for Excel, JSON, XML, HDF5, Parquet/Delta Lake, PySpark sessions, MongoDB, MS Access, PostgreSQL, SQL Server, CozoDB, Neo4j, Raphtory, Enterprise Architect, filesystem snapshots, and more.

Ontology modelling

from bnop.bnop_facades import BnopFacades
from bclearer_orchestration_services.identification_services.uuid_service.uuid_helpers.uuid_factory import (
    create_new_uuid,
)

repository_uuid = create_new_uuid()

product_type = BnopFacades.create_new_bnop_type(repository_uuid)
product = BnopFacades.create_bnop_object(
    object_uuid=create_new_uuid(),
    owning_repository_uuid=repository_uuid,
    presentation_name="Example Product",
)

BnopFacades.write_bnop_object_to_xml("bnop_snapshot.xml")

Use CKIDs from bclearer_core.ckids to classify tuples and relationships when you need richer BORO semantics.

Orchestration helpers

from bclearer_orchestration_services.b_app_runner_service.b_application_runner import run_b_application

def bootstrap():
    print("hello bCLEARer")

run_b_application(bootstrap)

Complement this with utilities from identification_services, log_environment_utility_service, static_code_analysis_service, and version_control_services to manage runtime behaviour and governance.

Testing & quality gates

pytest                                  # the default selection: the interop services suite, heavy tests deselected
pytest libraries/core/tests             # one package's suite
pytest -m "not heavy"                   # skip connectors that rely on external services
uv run ruff check .                     # lint
uv run ruff format --check .            # what CI checks

Tests live under libraries/*/tests. Heavy tests target databases or graph backends and are opt-in via the heavy marker. ruff is the only linter and the only formatter.

Documentation & next steps

• Architecture overview: documentation/bclearer_pipeline_framework.md
• Coding and testing standards: documentation/standards/
• Interop manual: documentation/emanual/

Contributing

Issues and feedback are welcome. Code pull requests are not being accepted yet, so please raise anything you find as an issue rather than as a patch.

For anything that does not fit an issue, contact support@ontoledgy.io.

Licence

This project is licensed under the GNU Affero General Public License v3.0 or later (AGPL-3.0-or-later). See LICENSE for the full text.

Commercial licences are available for organisations that cannot comply with the AGPL's network-use and source-disclosure terms. Contact support@ontoledgy.io for details.

Contact

support@ontoledgy.io