HKEx Filing Scraper

An open-source Python tool that scrapes 25+ years of Hong Kong Stock Exchange (HKEx) regulatory filings and ingests them into any combination of PostgreSQL, MySQL/MariaDB, SQLite, MongoDB, Neo4j, ClickHouse, DuckDB, and SurrealDB — with full-text extraction from PDF/HTML/Excel documents, structured tables, coverage tracking, and optional graph linking.

It uses the undocumented HKEx JSON API directly, which is significantly faster and more reliable than browser-based scraping.

Contents

• Database support
• Why this project
• How it works
• Features
• Installation
• Quick start
• Usage
• Configuration
• Database schema
• Documentation
• Development
• Contributing
• Roadmap
• License

Database support

Set DATABASE_TARGET to any ordered, comma-separated combination of these. Every sink is a first-class destination — the order only decides which one serves reads. The full matrix (licenses, capability differences, per-engine notes) is in docs/sinks; rows are in the documented popularity order.

Sink	Model	License	Extra	Idempotent upsert
postgres	relational	PostgreSQL License	postgres	ON CONFLICT DO UPDATE
mysql / mariadb	relational	GPLv2	mysql	ON DUPLICATE KEY UPDATE
sqlite	relational	Public domain	—	ON CONFLICT DO UPDATE
mongodb	document	SSPL¹	mongodb	update_one(upsert=True)
neo4j	graph	GPLv3 (Community)	neo4j	MERGE
clickhouse	columnar	Apache-2.0	clickhouse	ReplacingMergeTree + read-merge
duckdb	relational	MIT	duckdb	ON CONFLICT DO UPDATE
surrealdb	graph + document	BSL 1.1¹	—	UPSERT / RELATE

¹ Source-available, not OSI-approved — labelled exceptions per ADR 0003.

Why this project

Regulatory filings are the raw substrate for research, compliance, and LLM/RAG systems, but getting a complete, faithful, provenance-preserving copy is tedious: you have to reverse-engineer the HKEx API, handle a JSF session and pagination, parse Chinese/English bilingual PDFs, extract tables, and survive payload limits and database quirks. This tool does all of that and hands you a clean corpus.

The multi-sink design means you don't have to adopt a new database to use it: keep whichever store your team already runs, mirror everything into a second one for SQL/BI/dbt tooling, stream documents into a document store, or load a columnar engine for analytics. Set one variable (DATABASE_TARGET) and the same run feeds one or several sinks.

How it works

The scraper runs in two phases, builds one canonical record per filing, and hands each record to every sink listed in DATABASE_TARGET. Reads are served by the first configured sink that supports them.

flowchart LR
    A[HKEx JSON API] --> B[Phase 1: metadata]
    B --> C[Canonical record]
    C --> D{DATABASE_TARGET}
    D --> E[(PostgreSQL)]
    D --> F[(MySQL / MariaDB)]
    D --> G[(SQLite)]
    D --> H[(MongoDB)]
    D --> I[(Neo4j)]
    D --> J[(ClickHouse)]
    D --> K[(DuckDB)]
    D --> L[(SurrealDB)]
    B --> M[Graph linking]
    M --> D
    B --> N[Phase 2: download and extract]
    N --> C

• Phase 1 scrapes filing metadata from the undocumented HKEx JSON API through a JSF session, splitting the range into monthly chunks and deduplicating on a 16-character MD5 filingId.
• Graph linking writes has_filing and references_filing edges to every edge-capable sink when COMPANY_TABLE is set.
• Phase 2 downloads each filing's PDF/HTML/Excel document, extracts text and tables to Markdown, and writes the payload to every sink.
• Failure isolation means one sink's failure is logged and counted but never blocks another; the run exits non-zero if any configured sink failed.

More detail: Architecture and ADR 0002.

Features

• Fast API scraping — direct HKEx JSON API, no browser/Selenium.
• Full history — every filing from April 1999 to today, with chunk-level coverage verification.
• Document processing — downloads PDF/HTML/Excel and extracts full text plus structured tables (Markdown).
• Multi-sink — write to any combination of the nine sinks via DATABASE_TARGET (an ordered, comma-separated list). Each sink mirrors filings, documents, coverage, and edges with idempotent upserts, in its own native model.
• Graph linking — optional (company)-[has_filing]->(filing) and (filing)-[references_filing]->(company) edges on every edge-capable sink.
• Parallel and resumable — batching, parallel downloads, stalled-job detection, and per-chunk coverage tracking.
• Failure isolation — a failure on one sink never blocks or rolls back another; per-sink counters are reported every run, and the run exits non-zero if any configured sink failed.
• Optional dependencies — core is requests + beautifulsoup4; document extraction and every database driver (psycopg, PyMySQL, duckdb, pymongo, clickhouse-connect, neo4j) are extras with graceful fallback. SQLite needs no extra.

Installation

The package is published on PyPI:

pip install hkex-filing-scraper              # core; SQLite works out of the box
pip install "hkex-filing-scraper[all]"       # Excel extraction + dotenv + every database driver
pip install "hkex-filing-scraper[postgres]"  # add one sink driver at a time
pip install "hkex-filing-scraper[mysql]"     # MySQL and MariaDB
pip install "hkex-filing-scraper[duckdb]"    # or: mongodb, clickhouse, neo4j

To run the latest unreleased code, install straight from GitHub:

pip install "git+https://github.com/simonplmak-cloud/hkex-filing-scraper.git"

Optional extras: excel, postgres, mysql, duckdb, mongodb, clickhouse, neo4j, all, dev. SQLite and SurrealDB need no extra.

For a fully locked development environment, uv.lock pins every dependency including extras:

uv sync --frozen --all-extras

Releases carry signed build provenance and a CycloneDX SBOM — see Verifying a release.

# PDF text + table extraction. AGPL-3.0 — see the license note below.
pip install "hkex-filing-scraper[pdf]"

License note. The pdf extra installs PyMuPDF and pymupdf4llm, which are AGPL-3.0 (or a commercial license from Artifex). They are deliberately not part of all, so the default install stays permissive. If you distribute or host a service that includes them, the AGPL's network clause applies to you. See docs/legal.md.

Quick start

cp .env.example .env

Set DATABASE_TARGET to any ordered, comma-separated list of sink ids, then add that sink's connection settings. Every sink is written to; reads come from the first read-capable sink.

# PostgreSQL
DATABASE_TARGET=postgres
POSTGRES_DSN=postgresql://user:password@localhost:5432/hkex

# MySQL / MariaDB
DATABASE_TARGET=mysql
MYSQL_HOST=localhost  MYSQL_DATABASE=hkex  MYSQL_USER=hkex  MYSQL_PASSWORD=secret

# SQLite (no server)
DATABASE_TARGET=sqlite
SQLITE_PATH=hkex.db

# MongoDB
DATABASE_TARGET=mongodb
MONGODB_URI=mongodb://localhost:27017  MONGODB_DATABASE=hkex

# Neo4j
DATABASE_TARGET=neo4j
NEO4J_URI=bolt://localhost:7687  NEO4J_USER=neo4j  NEO4J_PASSWORD=secret

# ClickHouse
DATABASE_TARGET=clickhouse
CLICKHOUSE_HOST=localhost  CLICKHOUSE_DATABASE=hkex  CLICKHOUSE_USER=default

# DuckDB (no server)
DATABASE_TARGET=duckdb
DUCKDB_PATH=hkex.duckdb

# SurrealDB
DATABASE_TARGET=surrealdb
SURREAL_ENDPOINT=http://localhost:8000
SURREAL_PASSWORD=root

Then run:

hkex-scraper --metadata-only          # fast: metadata only
hkex-scraper                          # full: metadata + documents + graph

Several sinks at once

# Order sets read precedence.
DATABASE_TARGET=postgres,sqlite
POSTGRES_DSN=postgresql://user:password@localhost:5432/hkex
SQLITE_PATH=hkex.db

The schema is created automatically on startup — no manual DDL. See docs/sinks for the capability matrix. MongoDB (SSPL) and SurrealDB (BSL) are source-available, labelled exceptions.

Usage

# Last ~2 months (default)
hkex-scraper

# Full history from 1999
hkex-scraper --full-history

# Specific range
hkex-scraper --from-date 01/01/2024 --to-date 31/01/2024

# Metadata only / documents only / graphs only
hkex-scraper --metadata-only
hkex-scraper --backfill-docs
hkex-scraper --link-only

# Limit and dry-run
hkex-scraper --limit 500
hkex-scraper --dry-run

# Choose the sink(s) for this run (comma-separated, ordered)
hkex-scraper --database-target postgres
hkex-scraper --database-target sqlite
hkex-scraper --database-target postgres,sqlite

# Reporting
hkex-scraper --coverage-report
hkex-scraper --database-target postgres,sqlite --parity-report
hkex-scraper --database-target postgres,sqlite --verify

Command-line options

Flag	Description
--full-history	Scrape all filings from April 1999 to today.
--from-date DD/MM/YYYY	Start date for scraping.
--to-date DD/MM/YYYY	End date for scraping.
--limit N	Limit processing to N filings (0 = unlimited).
--metadata-only	Phase 1 only: metadata, no downloads.
--backfill-docs	Phase 2 only: download documents for existing filings.
--link-only	Only create/refresh graph edges.
--dry-run	Fetch but do not write to any database.
--database-target SINKS	Override DATABASE_TARGET (comma-separated; valid: postgres, mysql, sqlite, mongodb, mariadb, neo4j, clickhouse, duckdb, surrealdb).
--coverage-report	Print chunk coverage from the read source, then exit.
--parity-report	Print filing counts per sink and the spread, then exit.
--verify	Compare sinks by filing id and document hash, then exit non-zero on any difference.
--version	Print the version and exit.

Exit code is non-zero when any configured sink recorded write failures.

Configuration

Configuration is loaded from .env in the current working directory (not the project root). Full reference: docs/configuration.md.

Variable	Default	Purpose
DATABASE_TARGET	required	Comma-separated, ordered sink list: postgres, mysql, sqlite, mongodb, mariadb, neo4j, clickhouse, duckdb, surrealdb.
POSTGRES_DSN	—	Full PostgreSQL DSN (preferred).
POSTGRES_HOST / POSTGRES_PORT	localhost / 5432	Discrete connection (used when POSTGRES_DSN is empty).
POSTGRES_DATABASE / POSTGRES_USER / POSTGRES_PASSWORD	—	Discrete connection.
POSTGRES_SCHEMA	public	Schema for the mirrored tables.
POSTGRES_MIN_POOL / POSTGRES_MAX_POOL	1 / 15	Connection pool sizing.
MYSQL_DSN	—	Full MySQL DSN (preferred); MARIADB_DSN follows the same shape.
MYSQL_HOST / MYSQL_PORT / MYSQL_DATABASE / MYSQL_USER / MYSQL_PASSWORD	— / 3306 / —	MySQL connection; MARIADB_* falls back to these.
SQLITE_PATH	—	SQLite file path, or :memory:.
MONGODB_URI / MONGODB_DATABASE	—	MongoDB connection URI and database.
NEO4J_URI / NEO4J_USER / NEO4J_PASSWORD / NEO4J_DATABASE	—	Neo4j Bolt connection.
CLICKHOUSE_HOST / CLICKHOUSE_PORT / CLICKHOUSE_DATABASE / CLICKHOUSE_USER / CLICKHOUSE_PASSWORD	— / 8123 / —	ClickHouse connection.
DUCKDB_PATH	—	DuckDB file path, or :memory:.
SURREAL_ENDPOINT	—	SurrealDB HTTP endpoint (required when target includes surrealdb).
SURREAL_NAMESPACE / SURREAL_DATABASE	default	SurrealDB NS/DB.
SURREAL_USERNAME / SURREAL_PASSWORD	root / —	SurrealDB credentials.
COMPANY_TABLE	—	Company table; enables graph edges.
COMPANY_ID_PATTERN	{code}_{exchange}	Ticker → company key pattern.
MAX_DOWNLOAD_WORKERS	15	Parallel document downloads.
REQUEST_DELAY_SECONDS	0	Minimum seconds between request starts (pacing).

Database schema

Each sink stores the same four record families in its own native model. The schema is created automatically with idempotent DDL, and filing_id (16-char MD5) is the shared join key across all sinks.

Record	Relational (postgres, mysql, mariadb, sqlite, duckdb)	Key
Filing + document payload	exchange_filing	filing_id
Chunk coverage	scrape_coverage	(chunk_from, chunk_to, run_id)
Company → filing edges	has_filing	(company_id, filing_id)
Filing → referenced-company edges	references_filing	(filing_id, company_id)

Column shapes differ by engine: document_tables is jsonb (PostgreSQL), json (MySQL/MariaDB), JSON text (SQLite/DuckDB/ClickHouse), an embedded array (MongoDB), or a JSON string (Neo4j); referenced_tickers is text[] (PostgreSQL), json (MySQL), JSON text (SQLite/DuckDB), Array(String) (ClickHouse), an array (MongoDB), or a string array (Neo4j). SurrealDB uses SCHEMAFULL tables of the same names.

Writes are parameterised upserts, so re-running is idempotent. Document payloads use a separate statement, so a metadata-only re-run never overwrites extracted text or tables.

-- PostgreSQL examples; each sink guide has engine-specific queries.
SELECT company_ticker, count(*) FROM exchange_filing GROUP BY 1 ORDER BY 2 DESC LIMIT 10;

SELECT filing_id, title
FROM exchange_filing
WHERE referenced_tickers && ARRAY['0700.HK'];

SELECT filing_id, tbl->>'markdown'
FROM exchange_filing, jsonb_array_elements(document_tables) AS tbl
WHERE filing_type = 'Annual Report';

Documentation

• Getting started
• Docs site: https://simonplmak-cloud.github.io/hkex-filing-scraper/
• Database sinks (support matrix)
• PostgreSQL sink guide
• MySQL/MariaDB sink guide
• SQLite sink guide
• MongoDB sink guide
• Neo4j sink guide
• ClickHouse sink guide
• DuckDB sink guide
• SurrealDB sink guide
• Try it locally (examples/)
• Configuration reference
• CLI reference
• Architecture
• Troubleshooting
• Testing
• De-risking register
• Legal & Terms of Use
• Upgrading
• Releasing
• Release automation reference
• Documentation style guide
• Changelog

Development

pip install -e ".[dev,all]"
ruff check          # lint (py310, line-length 100)
ruff format --check # formatting
pytest              # unit tests (no DB/network required)

Tests are pure unit tests. SQLite and DuckDB contract tests run in-process on every pytest; integration tests that need a server are skipped unless that sink is configured.

Contributing

See CONTRIBUTING.md. Please report security issues per SECURITY.md. Ideas and questions are welcome in Discussions.

If this saves you time, a star helps others find it.

Roadmap

• Shipped — nine sinks on one contract: PostgreSQL, MySQL/MariaDB, SQLite, MongoDB, Neo4j, ClickHouse, DuckDB, SurrealDB.
• In progress — hardening: recorded HKEx API fixtures + canary, HTTP retry/backoff, --verify cross-sink reconciliation, fault-injection tests, SBOM/provenance (see docs/de-risking.md).
• Considered — OpenSearch, Cassandra, Valkey, TiDB (see ADR 0003).

Contributions that fit the roadmap, and good first issue items, are especially welcome.

License

MIT — see LICENSE. That covers this project's code only. Optional dependencies carry their own licenses — notably the pdf extra, which is AGPL-3.0; see docs/legal.md.

Data & Terms of Use: this is a research tool for the undocumented HKEx JSON API. Commercial redistribution of HKEx data may require a licensed HKEx feed; see docs/legal.md.