otb-cbso-webservice-pyclient

Download Belgian annual accounts from the National Bank's Central Balance Sheet Office (CBSO). Use the Python package or the cbso-fetch command.

The package downloads selected batches and files accounts by enterprise. Validation and repair run as separate operations. Downloads and repairs run sequentially. The package can also download missing PDFs for PDF-only filings and collect consolidated accounts.

Installation

Use Python 3.10 or later.

pip install otb-cbso-webservice-pyclient

Install the S3 extra for S3 destinations.

pip install 'otb-cbso-webservice-pyclient[s3]'

The core dependencies are urllib3 and fsspec. The S3 extra adds s3fs and its dependencies.

Configure subscriptions

Each CBSO product requires its own subscription key. Choose the products that match your subscriptions and the data you need.

Set the key for each product you want to download.

export CBSO_KEY_EXTRACTS='your-extracts-subscription-key'

Repair and PDF-only downloads can need individual-access subscriptions.

A batch request checks its product key. Reusing retained ZIP files requires no CBSO keys. Repair reports missing keys and continues with work that can use available subscriptions or saved data. Validation does not require CBSO subscription keys.

Command line

download requires dates, --products, and --destination. Supply past dates in YYYY-MM-DD form. Replace the example dates with the dates you need. Product availability determines which dates the service can supply. All other commands accept an input directory and scan files across all dates. validate and repair accept --formats; the default is json. validate-improved and repair-improved handle improved JSON only. repair-improved, download-pdf-only-accounts, and filter-consolidated-accounts also support --dry-run.

Download batches

download fetches references and accounts for each selected date and product. References connect deposit numbers to enterprise numbers. The command uses that connection to file accounts into enterprise folders.

Download one date as JSON-XBRL.

cbso-fetch download 2024-08-03 --products extracts --destination ./data

Download multiple dates and formats, with copies of the original ZIP files.

cbso-fetch download 2024-08-03 2024-08-04 \
  --products extracts \
  --destination ./data \
  --formats json xbrl pdf \
  --keep-zip

Select archive products directly to request archive batches. This example stores both authentic and improved JSON-XBRL accounts in S3.

cbso-fetch download 2024-08-03 \
  --products authentic-archive improved-archive \
  --destination s3://my-bucket/cbso

json selects JSON-XBRL account data. Reference files use ordinary JSON. Authentic products support all three formats. Improved products support only json. A command containing improved products must select only JSON. Use separate commands when you need other formats from authentic products.

Existing ZIP files under archives/{product}/{date}/ are reused without a batch request. Every run extracts the selected ZIP files, including retained copies. Existing output files are overwritten. Without retained ZIP files, repeated runs download the batches again. ZIP retention is off by default. Dates are processed in the supplied order. Entries without usable enterprise information remain under pending/ for inspection or repair. Downloading does not request individual accounts or automatically repair missing files.

Validate saved files

validate scans JSON files under references/ and checks the selected account formats. It uses reference contents to determine enterprise numbers, deposit numbers, and expected file paths. The scan includes all dates. It does not require download history.

cbso-fetch validate ./data --formats xbrl pdf

Omit --formats to check JSON only. Missing XBRL for PDF-only accounts appears under unavailable. Pending reference files and pending accounts in selected formats appear under unresolved.

Validation checks file presence. It does not inspect account contents or validate XBRL schemas. It makes no CBSO requests and leaves stored files unchanged. An S3 destination still requires access to its stored objects. A missing reference directory is an error unless matching pending files exist.

A reference scan cannot detect a deposit when both its reference and account files are absent. An empty reference directory produces an empty report. This does not confirm that all service data was downloaded. Run download again to restore missing references from batches.

Validate improved files

validate-improved scans JSON files under improved-references/. It checks the improvement named by each reference and reports unresolved pending improvements. For PDF-based deposits, it also checks OCR JSON, even without a separate OCR reference.

cbso-fetch validate-improved ./data
cbso-fetch validate-improved s3://my-bucket/annual-accounts

Improved accounts support JSON only, so this command has no --formats option. It makes no CBSO requests and changes no files. Run repair-improved to restore missing improved files.

Repair missing files

repair scans authentic references across all dates and restores selected account formats.

cbso-fetch repair ./data --formats xbrl pdf

It first resolves damaged references and missing enterprise numbers through individual reference requests. It then files pending accounts using saved or recovered references. The pending directory supplies each file's product and format. The filename supplies its deposit number. Repair preserves existing accounts and removes pending copies only after the destination file exists. It requests remaining account gaps individually. These requests require CBSO_KEY_AUTHENTIC.

The final report includes restored paths and any work that remains incomplete. Unreadable references, missing keys, and unavailable accounts remain visible in the report. Repair does not request XBRL for PDF-only accounts.

Repair does not download or extract batch ZIP files. Run download to extract retained ZIP files or download batches whose ZIP files were not retained. Use repair-improved for improved references and accounts.

Repair improved accounts across all dates

export CBSO_KEY_IMPROVED='your-improved-subscription-key'
cbso-fetch repair-improved ./data
cbso-fetch repair-improved s3://my-bucket/annual-accounts

repair-improved scans all JSON files beneath improved-references/, including enterprise subdirectories. It uses no date or product filters. It accepts camelCase and PascalCase reference fields.

It checks the improvement identified by each reference:

For models ending in -p, it also requests missing OCR JSON, even without a matching OCR reference. It does not infer additional corrections or euro conversions without their reference metadata.

Files go into improved-jsons/{enterprise}/. Existing files remain unchanged. Duplicate references cause at most one request per output path during a run. Processing is sequential. Individual requests use CBSO_KEY_IMPROVED. Use --env test for UAT2 and the same environment as the source references. S3 requires the optional s3 installation and storage credentials.

The command first resolves pending improvements and missing enterprise information. Individual improved-reference requests must match both the deposit number and improvement label. Recovered metadata goes into improved-references/{enterprise}/. Repair does not substitute authentic metadata for missing improved metadata.

Inferred OCR downloads create account files only. They do not create reference metadata or invent publication dates. Future validation and repair scans check those OCR files through the original PDF-based references. The command does not scan authentic references/.

Preview downloads without CBSO requests or file changes:

cbso-fetch repair-improved ./data --dry-run

The preview prints readable DOWNLOAD, EXISTS, and INVALID lines. Paths are relative to the input directory. No CBSO key is needed for a preview. Pending files appear as INVALID; the preview does not resolve their metadata or plan their recovery.

Use the same operation from Python:

from cbso_webservice_client import repair_improved

report = repair_improved("./data")
print(report.to_dict())

preview = repair_improved("./data", dry_run=True)
print(preview.planned)

The function also accepts environment="test" and a caller-owned client. Normal runs create and close a client when none is supplied. Dry runs create no client.

The report lists restored, existing, planned, missing, unavailable, and unresolved paths. Invalid references, missing enterprise numbers, missing keys, and HTTP 404 responses remain visible; processing continues. Normal CLI runs print JSON. Exit code 1 indicates incomplete work. Other service or storage failures stop processing and produce exit code 4. A missing improved-references/ directory is a storage failure unless pending improvements exist.

Download PDFs for PDF-only accounts

export CBSO_KEY_AUTHENTIC='your-authentic-subscription-key'
cbso-fetch download-pdf-only-accounts ./data
cbso-fetch download-pdf-only-accounts s3://my-bucket/annual-accounts

Preview the downloads without changing files:

cbso-fetch download-pdf-only-accounts ./data --dry-run

Dry runs print readable DOWNLOAD, EXISTS, and INVALID lines with paths relative to the input directory. They read references and check PDF locations, but make no CBSO requests or file changes. No CBSO subscription key is needed. S3 still requires storage access. A planned download does not confirm that CBSO has the PDF or that the destination permits writes. Invalid references produce exit code 1; planned downloads alone produce exit code 0.

This command scans all JSON files beneath references/, including enterprise subdirectories. It selects references whose model code ends in -p, which marks a PDF-based filing. It accepts camelCase and PascalCase fields. It uses no date or product filters.

Each missing PDF is requested individually through the authentic service and saved under pdfs/{enterpriseNumber}/{referenceNumber}.pdf. This is the same path used by PDF batch downloads. Existing PDFs remain unchanged. Duplicate references cause at most one request per output path during a run. The operation makes no batch requests and leaves references unchanged. Processing is sequential.

Use --env test for UAT2. Use the same environment as the source references. S3 requires the optional s3 installation and storage credentials. The authentic subscription key is needed only when a PDF request is necessary.

Use the library function with the same input directory:

from cbso_webservice_client import download_pdf_only_accounts

report = download_pdf_only_accounts("./data")
print(report.to_dict())

Pass dry_run=True to preview downloads from Python:

report = download_pdf_only_accounts("./data", dry_run=True)
print(report.planned)

Dry runs populate planned with output paths and leave written empty. Normal runs leave planned empty. The function also accepts environment="test" and an optional client=Client(...). The caller owns an explicitly supplied client. Otherwise, normal runs create and close their own client. Dry runs create no client.

The returned Report uses these fields:

• written: saved PDFs.
• skipped: PDFs already present.
• unresolved: invalid references or PDF-based references without a valid enterprise number.
• missing: PDFs that could not be downloaded because of missing keys or HTTP 404 responses.
• unavailable: PDFs for which CBSO returned HTTP 404.
• missing_keys: required subscription keys that were absent.

Invalid references, missing keys, and HTTP 404 responses do not stop processing other references. The CLI prints a JSON report. Exit code 1 indicates incomplete work; exit code 0 indicates completion. Other service or storage failures stop processing and produce exit code 4. A missing references/ directory is a storage failure.

Run this command before filter-consolidated-accounts. PDF-only downloads also check consolidated_accounts/BE{enterpriseNumber}_{referenceNumber}.pdf. If filtering has moved a PDF there, later downloads and dry runs report that path as skipped. Validation continues to use the formats recorded during batch downloads; this operation does not add PDF validation records.

Filter consolidated accounts

Run validation and any required repair before filtering. Filtering moves PDFs out of their original folders. Later validation can report them as missing, and repair can download them again.

cbso-fetch filter-consolidated-accounts ./data
cbso-fetch filter-consolidated-accounts s3://my-bucket/annual-accounts

Preview the copies and moves without changing files:

cbso-fetch filter-consolidated-accounts ./data --dry-run

Dry runs print readable COPY and MOVE lines with source and destination paths. Paths are relative to the input directory shown in the first line. EXISTS, MISSING, and INVALID lines report existing output PDFs, missing PDFs, and invalid references. Dry runs read references and check file locations. They do not create directories, write files, delete files, or read PDF contents. Exit codes follow the normal filtering rules.

This command scans all JSON files beneath references/, without date or product filters. It selects the fixed model codes m120-f-p, m121-f-p, m122-f-p, and m123-f-p. It accepts camelCase and PascalCase reference fields. S3 requires the optional s3 installation and storage credentials. CBSO keys are unnecessary.

For each selected reference, the command checks pdfs/{enterpriseNumber}/{referenceNumber}.pdf. When that PDF exists, it writes these files:

consolidated_accounts/BE{enterpriseNumber}_{referenceNumber}-reference.json
consolidated_accounts/BE{enterpriseNumber}_{referenceNumber}.pdf

The reference copy preserves the original bytes. Source references remain in place. Existing output files are overwritten. The source PDF is deleted only after the output PDF is saved successfully. Processing is sequential.

If the source PDF is absent but the output PDF exists, the command treats the PDF as complete. It still refreshes the output reference. If both PDFs are absent, the reference is copied to this error path:

consolidated_accounts_errors/BE{enterpriseNumber}_{referenceNumber}-reference.json

Existing error copies remain after later successful runs. Invalid JSON, missing required fields, and unsafe identifiers appear in the report. Processing continues with other references. Storage failures stop processing. A missing references/ directory is a storage failure.

The JSON report includes planned_copies, planned_moves, copied_references, and moved_pdfs. It also includes existing_pdfs, missing_pdfs, invalid_references, and complete. Missing PDFs alone produce exit code 0. Invalid references produce exit code 1; storage failures produce exit code 4. For this command, complete means no invalid references were found. Missing PDFs do not change that value.

Use the same operation from Python:

from cbso_webservice_client import filter_consolidated_accounts

report = filter_consolidated_accounts("./data")
print(report.to_dict())

The function accepts a local path or S3 URL and returns a ConsolidatedReport. Pass dry_run=True to preview transfers from Python:

report = filter_consolidated_accounts("./data", dry_run=True)
print(report.planned_copies)
print(report.planned_moves)

planned_copies and planned_moves map source paths to destination paths. Dry runs leave copied_references and moved_pdfs empty. Normal runs leave the planned transfer mappings empty. It does not download files.

Argument order and help

Options can precede positional dates. However, --products and --formats accept multiple values and can consume dates that follow them directly. Use -- to separate trailing dates from those options.

cbso-fetch download --destination ./data --products extracts -- 2024-08-03 2024-08-04

View command descriptions and options with --help.

cbso-fetch --help
cbso-fetch download --help
cbso-fetch validate --help
cbso-fetch validate-improved --help
cbso-fetch repair --help
cbso-fetch repair-improved --help
cbso-fetch download-pdf-only-accounts --help
cbso-fetch filter-consolidated-accounts --help

Environments and logging

Production is the default CBSO environment. Use --env test with download, repair, repair-improved, or download-pdf-only-accounts to select UAT2. Both validation commands and consolidated filtering have no CBSO environment option.

cbso-fetch download 2024-08-03 \
  --products extracts \
  --destination ./test-data \
  --env test

Configure subscription keys for the selected environment. Use separate destinations for production and test data. Use the same environment when repairing a destination.

Add -v or --verbose to enable informational logging. Use cbso-fetch --version to show the installed package version.

Python package

Import operations from cbso_webservice_client. download accepts one date string or a sequence of date strings. Pass products as a sequence of strings or Product enum members. The other operations accept an input directory. validate and repair also accept selected formats. repair_improved, download_pdf_only_accounts, and filter_consolidated_accounts accept an optional dry_run=True argument.

Download from Python

from cbso_webservice_client import Product, download

report = download(
    dates=["2024-08-03", "2024-08-04"],
    products=[Product.EXTRACTS],
    destination="./data",
    formats=["json", "pdf"],
    keep_zip=True,
)

print(report.written)
print(report.unresolved)

formats defaults to JSON. keep_zip defaults to False. destination accepts a local path or an S3 URL. Use environment="test" with download, repair, repair_improved, or download_pdf_only_accounts for UAT2.

Validate and repair from Python

These functions scan stored reference files. No earlier download process or saved format selection is required.

from cbso_webservice_client import repair, repair_improved, validate, validate_improved

validation = validate("./data", formats=["xbrl", "pdf"])
print(validation.to_dict())

repaired = repair("./data", formats=["xbrl", "pdf"])
print(repaired.to_dict())

improved_validation = validate_improved("./data")
improved_repaired = repair_improved("./data")

Validation never invokes repair. Call repair only when you want to restore files.

Supply keys directly

Use a Client context to pass keys in code and reuse its HTTP connections. Non-empty keys supplied to the client take precedence over environment variables.

from cbso_webservice_client import Client, download, repair

with Client(
    keys={
        "extracts": "your-extracts-subscription-key",
        "authentic": "your-authentic-subscription-key",
    },
    environment="production",
) as client:
    downloaded = download("2024-08-03", ["extracts"], "./data", client=client)
    repaired = repair("./data", formats=["json"], client=client)

An explicitly supplied client determines the request environment. The caller owns that client; the context closes it after use. Operations that make CBSO requests create and close their own clients when none is supplied. Directory dry runs create no CBSO client and do not use a supplied client.

Reports and errors

Downloads and repairs return Report; validation also returns Report. Consolidated filtering returns ConsolidatedReport, with the fields described in its command section. Both types provide to_dict() for JSON-compatible output.

complete describes the operation performed, not the entire destination. For dry runs, planned downloads do not make the report incomplete or confirm service availability. Report paths are relative to the input directory or destination. Use validate to check file presence, including files removed after an earlier download. A repair report can list a file as both missing and unavailable.

Library failures raise exceptions. These include MissingKeyError, HttpError, ValueError, storage errors, and urllib3 connection errors. Missing keys during individual repair and PDF-only requests appear in the report instead.

Completed CLI commands print a JSON report, including any incomplete work. Consolidated filtering, PDF-only downloads, and improved repair scans with --dry-run print readable text instead. The filtering command treats missing PDFs as informational; its exit behaviour is described below. Command failures write an error message to standard error.

Storage and repeated runs

Account and reference paths include the enterprise number and deposit number. Improved filenames also identify the improvement type.

references/0203201340/2024-123.json
jsons/0203201340/2024-123.json
pdfs/0203201340/2024-123.pdf
improved-jsons/0203201340/2024-123-aa-corrected.json
archives/extracts/2024-08-03/json.zip
pending/extracts/2024-08-03/json/2024-456.json

Commands do not read or write _state/. Existing contents can remain; they have no effect on operations. References and pending files provide the metadata needed for validation and repair.

A repeated download reuses each ZIP file that exists at its expected archive path. It extracts retained ZIP files again, restoring their references and accounts. Remove a retained ZIP file if you need to fetch that batch again from CBSO. Without --keep-zip, newly fetched ZIP files are not saved.

Before 05:00 Belgian time on the following day, HTTP 404 and empty successful responses appear under missing_batches. After that time, the downloader treats either response as an empty batch for that run. Empty responses are not saved as ZIP files. A later run requests those batches again. Non-empty invalid ZIP data remains an error. A retained invalid ZIP must be replaced before extraction can succeed.

For S3, permit object reads and writes beneath the destination prefix. Permit object deletion for pending-file recovery and consolidated PDF moves. Permit bucket listing for storage inspection and directory scans. Storage permission failures remain errors; they never count as absent files.

Development

See CONTRIBUTING.md for setup, tests, coverage, and release instructions.

Licence

Proprietary. © openthebox.