goosefs-metastore-client 0.1.1

A Python client for connecting to GooseFS Table Master via gRPC

GooseFS Metastore Client

A Python client library for connecting to GooseFS Table Master service via gRPC protocol.

Overview

GooseFS Metastore Client provides a Python interface to interact with GooseFS's Table Master service, enabling you to manage databases and tables in the GooseFS catalog.

Architecture

The client follows this call flow:

Python Client (GoosefsMetastoreClient)
    ↓
gRPC Client (TableMasterClientServiceStub)
    ↓
[gRPC Network Call]
    ↓
GooseFS Master (TableMasterClientServiceHandler)
    ↓
DefaultTableMaster

Features

• List and retrieve databases and tables from GooseFS catalog
• Attach external databases (e.g., from Hive) to GooseFS
• Mount/unmount tables for caching in GooseFS
• Synchronize databases with underlying metadata stores
• Retrieve table access statistics
• Get table and partition column statistics
• Transform tables with custom definitions

Installation

pip install -e .

Requirements

• Python >= 3.7
• grpcio >= 1.50.0
• protobuf >= 3.20.0
• A running GooseFS cluster with Table Master service enabled

Quick Start

Basic Usage

from goosefs_metastore_client import GoosefsMetastoreClient

GOOSEFS_HOST = "localhost"
GOOSEFS_PORT = 9200

with GoosefsMetastoreClient(GOOSEFS_HOST, GOOSEFS_PORT) as client:
    databases = client.get_all_databases()
    for db_info in databases:
        print(f"Database: {db_info.name}")

List Databases

from goosefs_metastore_client import GoosefsMetastoreClient

with GoosefsMetastoreClient("localhost", 9200) as client:
    databases = client.get_all_databases()
    for db in databases:
        print(f"{db.name} - Type: {db.type}")

Get Database Details

from goosefs_metastore_client import GoosefsMetastoreClient

with GoosefsMetastoreClient("localhost", 9200) as client:
    database = client.get_database("my_database")
    print(f"Location: {database.location}")
    print(f"Owner: {database.owner_name}")

Attach External Database

from goosefs_metastore_client import GoosefsMetastoreClient

with GoosefsMetastoreClient("localhost", 9200) as client:
    sync_status = client.attach_database(
        udb_type="hive",
        udb_db_name="hive_db",
        db_name="goosefs_db",
        configuration={
            "hive.metastore.uris": "thrift://hive-metastore:9083",
        },
        auto_mount=True,
    )
    print(f"Tables synced: {len(sync_status.tables_updated)}")

List and Get Tables

from goosefs_metastore_client import GoosefsMetastoreClient

with GoosefsMetastoreClient("localhost", 9200) as client:
    tables = client.get_all_tables("my_database")
    for table in tables:
        print(f"Table: {table.name}, Mounted: {table.is_mount}")
    
    table_info = client.get_table("my_database", "my_table")
    print(f"Owner: {table_info.owner}")
    for col in table_info.schema.cols:
        print(f"Column: {col.name} ({col.type})")

Mount/Unmount Tables

from goosefs_metastore_client import GoosefsMetastoreClient

with GoosefsMetastoreClient("localhost", 9200) as client:
    client.mount_table("my_database", "my_table")
    print("Table mounted for caching")
    
    client.unmount_table("my_database", "my_table")
    print("Table unmounted")

Access Statistics

from goosefs_metastore_client import GoosefsMetastoreClient

with GoosefsMetastoreClient("localhost", 9200) as client:
    stats = client.access_stat(days=7, top_nums=10)
    for stat in stats:
        print(f"{stat.db_name}.{stat.tb_name}: {stat.hots} accesses")

Using Builders

The client provides builder classes to construct database and table objects:

from goosefs_metastore_client.builders import DatabaseBuilder, TableBuilder, FieldSchemaBuilder

database = DatabaseBuilder(
    db_name="my_database",
    description="My test database",
    location="/user/hive/warehouse/my_database.db",
    owner_name="admin",
    parameters={"key": "value"},
).build()

API Reference

GoosefsMetastoreClient

Main client class for interacting with GooseFS Table Master.

Database Operations

• get_all_databases() - Get all databases in the catalog
• get_database(db_name) - Get a specific database by name
• attach_database(udb_type, udb_db_name, db_name, ...) - Attach an external database
• detach_database(db_name) - Detach a database from the catalog
• sync_database(db_name) - Sync a database with its underlying store

Table Operations

• get_all_tables(database) - Get all tables in a database
• get_table(db_name, table_name) - Get a specific table
• mount_table(db_name, tb_name) - Mount a table to GooseFS
• unmount_table(db_name, tb_name) - Unmount a table from GooseFS

Statistics and Analytics

• access_stat(days, top_nums) - Get table access statistics
• get_table_column_statistics(db_name, table_name, col_names) - Get column statistics
• get_partition_column_statistics(db_name, table_name, col_names, part_names) - Get partition statistics

Advanced Operations

• read_table(db_name, table_name, constraint) - Read table partitions with constraints
• transform_table(db_name, table_name, definition) - Transform a table
• get_transform_job_info(job_id) - Get transformation job information

Configuration

Connection Parameters

• host: GooseFS master hostname
• port: Table Master client service port (default: 9200)
• max_retries: Maximum retry attempts for failed requests (default: 3)
• timeout: Timeout in seconds for gRPC calls (default: 30)
• credentials: Optional gRPC credentials for secure connections

Example with Custom Configuration

client = GoosefsMetastoreClient(
    host="goosefs-master.example.com",
    port=9200,
    max_retries=5,
    timeout=60,
)
client.connect()
try:
    databases = client.get_all_databases()
finally:
    client.close()

Development

Setup Development Environment

pip install -r requirements.dev.txt

Running Tests

pytest tests/

Code Formatting

black goosefs_metastore_client/

Linting

flake8 goosefs_metastore_client/
mypy goosefs_metastore_client/

Project Structure

goosefs-metastore-client/
├── goosefs_metastore_client/
│   ├── __init__.py
│   ├── goosefs_metastore_client.py
│   └── builders/
│       ├── __init__.py
│       ├── abstract_builder.py
│       ├── database_builder.py
│       ├── field_schema_builder.py
│       └── table_builder.py
├── grpc_files/
│   ├── __init__.py
│   ├── common_pb2.py
│   ├── job_master_pb2.py
│   ├── table_master_pb2.py
│   ├── table_master_pb2_grpc.py
│   └── proto/
│       ├── common.proto
│       ├── job_master.proto
│       └── table_master.proto
├── examples/
│   ├── list_databases.py
│   ├── get_database.py
│   ├── attach_database.py
│   ├── list_tables.py
│   ├── get_table.py
│   ├── mount_table.py
│   └── access_statistics.py
├── tests/
│   └── unit/
│       └── goosefs_metastore_client/
│           └── builders/
├── setup.py
├── requirements.txt
└── README.md

Comparison with Hive Metastore Client

Feature	Hive Metastore Client	GooseFS Metastore Client
Protocol	Thrift	gRPC
Server	Hive Metastore	GooseFS Table Master
Connection	TSocket + TBinaryProtocol	gRPC Channel + Stub
Retry Logic	Manual	Built-in with configurable retries
Database Operations	Thrift API	gRPC API
Table Operations	Thrift API	gRPC API

License

Apache License 2.0

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.