datus-doris

Apache Doris database adapter for Datus.

Features

• MySQL-protocol connection and SQL execution
• Multi-catalog discovery and context switching
• Catalog-aware table, view, schema, and sample-row metadata
• Asynchronous materialized-view discovery and DDL retrieval
• Three-part identifiers: catalog.database.table
• Doris migration capability, table-layout, and type-mapping guidance
• List, CSV, Pandas, and Arrow result formats inherited from datus-mysql

Installation

pip install datus-doris

The package installs datus-db-core and datus-mysql as dependencies and registers the doris adapter through the datus.adapters entry-point group.

Configuration

database:
  type: doris
  host: localhost
  port: 9030
  username: root
  password: ""
  catalog: internal
  database: analytics

The built-in catalog is internal. External catalogs, including Hive Metastore catalogs, can be selected through the same connector API.

Python API

from datus_doris import DorisConfig, DorisConnector

config = DorisConfig(
    host="localhost",
    port=9030,
    username="root",
    password="",
    catalog="internal",
    database="analytics",
)

with DorisConnector(config) as connector:
    catalogs = connector.get_catalogs()
    tables = connector.get_tables(database_name="analytics")
    views = connector.get_views(database_name="analytics")
    materialized_views = connector.get_materialized_views_with_ddl(
        database_name="analytics"
    )
    result = connector.execute_query("SELECT 1")

Catalog context can be switched explicitly:

connector.switch_catalog("hive_catalog")
databases = connector.get_databases()
connector.switch_catalog("internal")

Fully qualified identifiers are rendered as Doris three-part names:

connector.full_name(
    catalog_name="internal",
    database_name="analytics",
    table_name="events",
)
# `internal`.`analytics`.`events`

Development and testing

Unit tests do not require a database:

ci/run-unit-tests.sh datus-doris

The repository integration runner starts Apache Doris 4.1.3 (FE and BE in one apache/doris:all-in-one-4.1.3 container) and Hive Metastore 4.0.1, waits for an alive backend and a successful OLAP DDL probe, runs the complete integration suite, and removes the test services afterward:

ci/run-integration-tests.sh doris

To run the package-local workflow manually:

cd datus-doris
docker compose up -d --wait
./scripts/test.sh integration
docker compose down -v

The integration suite covers catalog operations, Hive external catalogs, tables, views, asynchronous materialized views, DDL/DML, sample rows, TPC-H queries, and list/CSV/Pandas/Arrow output.

Environment variables

The apache/doris:all-in-one-4.1.3 image carries its own integration-test FE and BE configuration, so there are no heap or replication knobs to set here. FE_CONFIG_EXTRA and BE_CONFIG_EXTRA are appended to fe.conf / be.conf at startup if a single setting ever needs overriding.

License

Apache License 2.0