fd-open-data-protocol

English | 中文

The open-data datasource protocol: a manifest contract a datasource exposes (datasource + functions + columns + concept hints + fetch reference) so that fd-open-data-mcp - or any consumer - can ingest it via register_datasource.

Ship one manifest file -> the datasource is added. No consumer-side wiring.

One-click install

This library is a dependency of fd-open-data-mcp (pulled in transitively). To install the entire finddata stack (hub + every datasource + ontology DB):

pip install "fd-open-data-mcp[data]" fd-polygon fd-cn-report

fd-open-data-mcp migrate \
  && fd-open-data-mcp import-catalog \
  && fd-open-data-mcp consume-concepts \
  && fd-open-data-mcp propose-bindings \
  && fd-open-data-mcp seed-entities \
  && fd-open-data-mcp generate-schedules \
  && fd-open-data-mcp register-discovered

fd-open-data-mcp serve

The manifest

A YAML/JSON file (or a Python module exposing CATALOG):

version: "1"
name: my-source
label: My Source
ranking_seed: [0.7, 0.7]            # [quality, accessibility] heuristic seed
functions:
  - command: get_data
    frequency: daily
    parameters: [{name: symbol, type: str, required: true}]
    columns:
      - {name: close, type: float, frequency: daily}
concepts:                           # column -> concept hints (measure/entity_type here)
  - {column: close, concept: price.close, entity_type: stock, unit: currency, frequency: daily}
fetch:
  runner: my-source                  # built-in runner name, OR module: "pkg.mod:run"

See examples/example_stock.yaml (declarative) and examples/example_macro.py (a DataProvider class with run()).

Load + validate

from fd_open_data_protocol.loader import load_catalog
manifest = load_catalog("examples/example_stock.yaml")
print(manifest.name, len(manifest.functions))

load_catalog accepts a YAML/JSON file path, a .py file exposing CATALOG, a "pkg.mod" module path, or a dict.

Register with fd-open-data-mcp

fd-open-data-mcp register-datasource examples/example_stock.yaml

or the MCP tool register_datasource(path).

Publish a datasource from another project

In your datasource package's pyproject.toml:

[project.entry-points."fd_open_data_mcp.datasources"]
my-source = "my_pkg.catalog:CATALOG"

pip install my-pkg -> fd-open-data-mcp auto-registers it on import_catalog.

Schema

• DatasourceManifest: name, label, source_url, scanner_mode, ranking_seed, functions[], concepts[], entities[], entity_definitions[], relationships[], fetch.
• FunctionSpec: command, category, description, parameters[], columns[], frequency, verified.
• ColumnSpec: name, type, description, meaning, semantic_type, frequency + datasource (column-level).
• ConceptHint: column, concept, entity_type, measure, unit, frequency, confidence, concept_family (optional).
• EntitySpec: entity_type, coverage ("universe"|"explicit"), codes[] (for explicit coverage).
• Entity: entity_type, code, name_en, name_zh, metadata{}, relationships[].
• EntityRelationship: target_entity_type, target_code, relation_type, confidence, metadata{}.
• RelationshipSpec: relation_type, source_entity_type, target_entity_type, resolver_module.
• FetchRef: runner (built-in name) | module ("pkg.mod:func").

measure + entity_type are concept-level (disambiguate GDP-nominal vs GDP-PPP; stock close vs fund NAV). Column-level frequency/datasource support composite functions whose columns come from different sources at different cadences.

Entity Definitions

The protocol supports two ways to declare entities:

1. Coverage Declaration (entities[])

Declares which entity types the datasource covers:

entities:
  - entity_type: stock
    coverage: explicit
    codes: [AAPL, MSFT, GOOGL]

• coverage: "universe" - datasource can fetch data for all entities of this type
• coverage: "explicit" - datasource only covers the listed codes

2. Entity Metadata (entity_definitions[])

Defines canonical entity metadata (names, attributes, relationships):

entity_definitions:
  - entity_type: stock
    code: AAPL
    name_en: Apple Inc.
    name_zh: 苹果公司
    metadata:
      exchange: NASDAQ
      sector: Technology
    relationships:
      - target_entity_type: industry
        target_code: gics_10
        relation_type: belongs_to

When included, entities are registered in the ontology database during register_datasource().

Reserved metadata key: external_ids

An entity definition may anchor itself in an external entity graph under the reserved metadata key external_ids. Registration persists each entry as a per-source identifier, so consumers can resolve the external id back to the local entity.

entity_definitions:
  - entity_type: country
    code: CN
    name_en: China
    metadata:
      external_ids:
        wikidata: Q148          # Wikidata QID (Q followed by digits)
        datacommons: country/CHN  # Data Commons DCID

Supported anchor sources are wikidata (QID, validated as Q<digits>) and datacommons (non-empty DCID). A malformed or unknown anchor is logged and skipped — it never fails registration.

Canonical Entity Types

The entity-type vocabulary is defined by fd_open_data_protocol/vocabulary/entity_types.yaml (14 types) and is the authority for manifest validation:

Type	Description	Example IDs
country	ISO codes	CN, US, JP
city	Municipalities	beijing, shanghai
stock	A-shares	600000.SH, 000001.SZ
fund	ETFs/funds	etf_code, fund_code
bond	Bonds	bond_code
index	Indices	SH000001, SZ399001
future	Futures	cu2412, rb2401
crypto	Cryptocurrencies	btc, eth
organization	General orgs	org_code
industry	Classifications	shenwan_1_01, gics_10
exchange	Exchanges / venues	XNYS, XNAS, SSE
company	Public companies	AAPL, TSLA
commodity	Raw goods	steel, oil, grain
person	Humans (fund managers)	person_code

Semantic vocabulary

The protocol ships the publishable semantic core of FindData as versioned YAML under fd_open_data_protocol/vocabulary/:

File	Registry
entity_types.yaml	entity types (the manifest vocabulary above)
concepts.yaml	concept families + their seeded Variables
qualifiers.yaml	qualifiers (nominal_current, real_constant, ppp, per_capita, growth, ...)
units.yaml	units of measure (UN/CEFACT codes + ISO 4217 currencies)
relation_types.yaml	entity relationship types
event_types.yaml	event types (empty in this release)
crosswalks/*.yaml	Variable ↔ external-vocabulary assertions

The vocabulary files ship as package data (0.3.0+), so load_vocabulary() works from an installed wheel with no checkout on disk.

from fd_open_data_protocol.vocabulary import load_vocabulary, validate_vocabulary

vocab = load_vocabulary()                    # shipped files
vocab = load_vocabulary("/path/to/dir")      # or an explicit directory
vocab.version                                # release version of the file set
vocab.get("concept", "GDP").uri              # https://schema.finddata.tech/concept/GDP
vocab.entity_type_ids()                      # the manifest entity-type vocabulary
vocab.seeded_variables()                     # Variables a family declares
errors = validate_vocabulary()               # [] when the file set is valid

Every entry carries a stable URI of the form https://schema.finddata.tech/<kind>/<Id>, where <kind> is one of entity-type, concept, qualifier, unit, relation-type, event-type.

Stability rules. A URI is never reassigned to a different meaning and entries are never deleted. Retirement is expressed with a superseded_by pointer to the successor entry, which keeps the URI resolvable. The file set carries one release version, and the loader rejects a file set whose files disagree.

Concept families and Variables

Concepts are two-level. A concept family (fd:GDP, fd:Population) is the small curated vocabulary from concepts.yaml; a Variable is the qualified combination (code, entity_type, measure, unit, frequency) — the identity a datasource actually binds columns to.

A manifest concept hint may point at its family with the optional concept_family field, which the loader validates against concepts.yaml:

concepts:
  - {column: close, concept: price.close, entity_type: stock, unit: currency,
     frequency: daily, concept_family: PriceClose}

Omitting it preserves the previous behaviour — no family validation occurs, and the consumer derives a family from the concept code.

Manifest Declaration Requirement

Every fd- datasource package MUST declare a DatasourceManifest via one of the following mechanisms:*

1. Python module: Expose a CATALOG dict in a module (e.g., catalog.py) that conforms to the DatasourceManifest schema
2. YAML/JSON file: Place a manifest file at the package root (e.g., catalog.yaml or catalog.json)
3. Entry-point declaration: Register the manifest path in pyproject.toml under [project.entry-points."fd_open_data_mcp.datasources"]

The declaration SHALL be discoverable by fd-open-data-mcp's auto-discovery mechanism (register-discovered command). Packages without a CATALOG declaration SHALL NOT be considered compliant with the fd-open-data-protocol.

Recommended Package Structure

my-datasource/
├── pyproject.toml          # declares entry-point
└── my_pkg/
    ├── __init__.py
    └── catalog.py          # exposes CATALOG = { ... }

Entry-Point Declaration

In your pyproject.toml:

[project.entry-points."fd_open_data_mcp.datasources"]
my-source = "my_pkg.catalog:CATALOG"

After pip install my-pkg, the package is automatically discoverable:

fd-open-data-mcp register-discovered

Auto-Discovery Flow

1. Install package → pip install my-datasource
2. Entry-point registered → setuptools records my-source = "my_pkg.catalog:CATALOG"
3. Auto-discover → fd-open-data-mcp register-discovered scans all entry-points
4. Load manifest → load_catalog() validates and parses the CATALOG dict
5. Register to ontology → register_datasource() upserts sources/functions/columns/concepts

Compliance Checklist

Before publishing a new datasource package, ensure:

• Package exposes a CATALOG dict or manifest file
• pyproject.toml declares entry-point under fd_open_data_mcp.datasources group
• CATALOG conforms to DatasourceManifest schema (version, name, label, functions[], concepts[], fetch)
• load_catalog() can successfully parse the manifest
• fd-open-data-mcp register-discovered discovers and registers the package

Working Examples

• fd-world: fd_world/catalog.py + entry-point in pyproject.toml
• fd-cn-gov: fd_cn_gov/catalog.py + entry-point in pyproject.toml
• fd-cn-report: catalog.py + entry-point in pyproject.toml

Template

Copy template/datasource.template.yaml (declarative) or template/provider_template.py (a BaseDataProvider class with run()).