fd-cn-gov

Scrapers + a self-contained datasource registry for Chinese central-government ministry open-information archives. Catalog-crawls the public notice / news / data archives of 11 ministries (MOF, PBC, NDRC, MOFCOM, MOHURD, MOT, MOA, SAFE, MNR, MEE, MEM), emitting one JSON record per listed document, and ships a SQLite + JSON registry describing every datasource and its column schema.

Standalone: no monorepo, no MCP server, no daas.db dependency. The MANIFEST at the top of each scraper is the single source of truth; the bundled registry.db / registry.json are derived artifacts.

Ministries

Each scraper emits a JSON array of records to stdout with at least title, date (YYYY-MM-DD, from the URL t<YYYYMMDD>_ token with a <span> fallback), and url (absolute, the primary key). Some add section / subsection / doc_type / department. See fd-cn-gov describe <name> for the exact columns of any source.

Install

pip install fd-cn-gov

Requires Python ≥3.10. Dependencies: scrapling (HTTP + adaptive parsing), sqlalchemy.

From source

pip install git+https://github.com/FindDataOfficial/cn-goverment-datasource.git

CLI

# List the 11 registered sources
fd-cn-gov list

# Show one source's identity + full column schema
fd-cn-gov describe mof_gkml_archive

# Crawl one archive (default: 50 pages per sub-archive; prints JSON to stdout)
fd-cn-gov crawl mof_gkml_archive --max-pages 2 > records.json

# Full crawl, no page cap (use sparingly — see Polite crawling below)
fd-cn-gov crawl mof_gkml_archive --all > records.json

# Regenerate the bundled registry from the scripts' MANIFESTs
fd-cn-gov build-registry

Python API

import fd_cn_gov

# Discover datasources
for s in fd_cn_gov.list_sources():
    print(s.name, s.label, s.url)

# One source + its column schema
src = fd_cn_gov.get_source("mof_gkml_archive")
cols = fd_cn_gov.get_columns("mof_gkml_archive")  # -> list[Column]
#   Column(name='url', type='string', primary_key=True, nullable=False,
#          description='absolute document URL (.htm/.html/.pdf)',
#          source_field='a@href', semantic_type='url', ...)

Registry schema

The bundled fd_cn_gov/registry/registry.db is a 3-table SQLite, mirroring the column shapes of the originating daas.db for these sources (no foreign keys, no stale-FK footgun):

• sources — one row per ministry: id, name, label, url, description, category, category_label, config_json
• datasource_columns — one row per output column: datasource_id, table_name, column_name, column_type, is_primary_key, is_nullable, description, source_field, unit, semantic_type
• scraw_configs — one row per scraper: name, url, columns_json

registry.json is a deterministic full dump of the same three tables (sort_keys), so the registry is diff-friendly in code review.

To regenerate after editing a MANIFEST:

fd-cn-gov build-registry

The build is logically idempotent: re-running produces an identical .dump and a byte-identical registry.json. (Only SQLite's file-change-counter header byte differs on each write — unavoidable on any SQLite write; compare via .dump or registry.json for equality.)

Polite crawling

These are .gov.cn hosts. Each scraper paces requests (SLEEP ≈ 0.3s between pages) and defaults to a 50-page cap per sub-archive. Use --max-pages N to bound a crawl further; reserve --all for when you genuinely need full history and can afford the time. Do not run multiple --all crawls in parallel against the same ministry.

Project layout

fd-cn-gov/
├── pyproject.toml
├── README.md
├── LICENSE
└── fd_cn_gov/
    ├── __init__.py            # public read API: list_sources / get_source / get_columns
    ├── cli.py                 # fd-cn-gov CLI
    ├── scraw_contract.py      # ScrawManifest / ScrawColumn dataclasses (vendored, trimmed)
    ├── build_registry.py      # regenerates registry.db + registry.json from MANIFESTs
    ├── scripts/               # 11 ministry scrapers, each with a module-level MANIFEST
    │   ├── mof_gkml_archive.py
    │   └── ...
    └── registry/
        ├── __init__.py        # read API over the bundled DB
        ├── registry.db        # generated — checked in
        └── registry.json      # generated — checked in

License

MIT