datus-odps

ODPS (MaxCompute) database adapter for Datus.

Installation

pip install datus-odps

This will automatically install the required dependencies:

• datus-agent
• pyodps
• pandas
• pyarrow

Usage

The adapter is automatically registered with Datus when installed. Configure your database connection in your Datus configuration:

database:
  type: odps
  access_id: your_access_key_id
  access_key: your_access_key_secret
  project: your_project_name
  endpoint: https://service.cn-shanghai.maxcompute.aliyun.com/api

Or use programmatically:

from datus_odps import ODPSConnector, ODPSConfig

# Create connector with config object
config = ODPSConfig(
    access_id="your_access_key_id",
    access_key="your_access_key_secret",
    project="your_project_name",
    endpoint="https://service.cn-shanghai.maxcompute.aliyun.com/api"
)
connector = ODPSConnector(config=config)

# Or create connector with dict
connector = ODPSConnector(config={
    "access_id": "your_access_key_id",
    "access_key": "your_access_key_secret",
    "project": "your_project_name",
    "endpoint": "https://service.cn-shanghai.maxcompute.aliyun.com/api"
})

# Test connection
connector.test_connection()

# Execute query
result = connector.execute_query("SELECT * FROM your_table LIMIT 10")
print(result.sql_return)

# Get table list
tables = connector.get_tables()
print(f"Tables: {tables}")

# Get table schema
schema = connector.get_schema(table_name="your_table")
for column in schema:
    print(f"{column['name']}: {column['type']}")

# Get tables with DDL
tables_with_ddl = connector.get_tables_with_ddl()
for table_info in tables_with_ddl:
    print(f"Table: {table_info['table_name']}")
    print(f"DDL: {table_info['definition']}")

# Get sample rows
samples = connector.get_sample_rows(tables=["your_table"], top_n=5)
for sample in samples:
    print(f"Sample from {sample['table_name']}:")
    print(sample['sample_rows'])

Configuration Options

The ODPSConfig class supports the following parameters:

• access_id (required): Aliyun Access Key ID
• access_key (required): Aliyun Access Key Secret
• project (required): ODPS project name
• endpoint (required): ODPS service endpoint (e.g., https://service.cn-shanghai.maxcompute.aliyun.com/api)
• timeout_seconds (optional): Connection timeout in seconds (default: 30)
• use_sqa (optional): Whether to use SQA (SQL Query Acceleration)
• quota_name (optional): Quota name for resource management

Features

• Full CRUD operations (SELECT, INSERT, UPDATE, DELETE)
• DDL execution (CREATE, ALTER, DROP)
• Metadata retrieval (tables, schemas)
• Sample data extraction
• Multiple result formats (pandas, arrow, csv, list)
• Connection management
• Comprehensive error handling
• Project-based namespace support (ODPS uses projects as databases)

ODPS-Specific Notes

• Project as Database: ODPS uses projects as the main namespace (similar to databases in traditional SQL databases)
• Table Naming: Tables are referenced as project.table_name
• Schemas: ODPS may support schemas in newer versions, but typically uses projects as the main namespace
• SQL Execution: All SQL queries are executed through PyODPS SDK

Requirements

• Python >= 3.12
• ODPS (MaxCompute) account with valid Access Key credentials
• datus-agent >= 0.2.1
• pyodps >= 0.11.0
• pandas >= 2.0.0
• pyarrow >= 10.0.0

License

Apache License 2.0