airbook-duckdb-utils-vb 1.0.0

Utility functions for DuckDB to BigQuery data type mapping and other database operations

Airbook DuckDB Utils

A Python package providing utility functions for DuckDB to BigQuery data type mapping and other database operations.

Installation

From Private PyPI (Recommended)

pip install airbook-duckdb-utils --extra-index-url https://your-private-pypi-server.com/simple/ --trusted-host your-private-pypi-server.com

From Source

git clone https://github.com/airbook/airbook-duckdb-utils.git
cd airbook-duckdb-utils
pip install -e .

Usage

Basic Type Mapping

from airbook_duckdb_utils import map_duckdb_data_type

# Map individual types
bigquery_type = map_duckdb_data_type('INTEGER')
print(bigquery_type)  # Output: NUMERIC

bigquery_type = map_duckdb_data_type('VARCHAR')
print(bigquery_type)  # Output: STRING

bigquery_type = map_duckdb_data_type('DECIMAL(18,3)')
print(bigquery_type)  # Output: NUMERIC

Advanced Usage

from airbook_duckdb_utils.type_mapping import (
    map_duckdb_data_type,
    get_type_mapping_dict,
    is_supported_type
)

# Check if a type is supported
if is_supported_type('TIMESTAMP'):
    mapped_type = map_duckdb_data_type('TIMESTAMP')
    print(f"TIMESTAMP maps to {mapped_type}")

# Get all supported mappings
all_mappings = get_type_mapping_dict()
print(f"Total supported types: {len(all_mappings)}")

# Example: Process a schema
schema = [
    ('id', 'BIGINT'),
    ('name', 'VARCHAR'),
    ('created_at', 'TIMESTAMP'),
    ('metadata', 'JSON'),
    ('score', 'DECIMAL(10,2)')
]

bigquery_schema = []
for column_name, duckdb_type in schema:
    bq_type = map_duckdb_data_type(duckdb_type)
    bigquery_schema.append((column_name, bq_type))
    
print("BigQuery Schema:")
for col_name, col_type in bigquery_schema:
    print(f"  {col_name}: {col_type}")

Supported Type Mappings

DuckDB Type	BigQuery Type	Notes
INTEGER, BIGINT, etc.	NUMERIC	All integer variants
FLOAT, DOUBLE	FLOAT64	Floating point numbers
VARCHAR, TEXT	STRING	Text data
BOOLEAN	BOOLEAN	Boolean values
DATE	DATE	Date values
TIMESTAMP	DATETIME	Timestamp without timezone
TIMESTAMPTZ	TIMESTAMP	Timestamp with timezone
JSON	JSON	JSON data
ARRAY, LIST	JSON	Arrays stored as JSON
STRUCT	JSON	Structured data as JSON
BLOB	BYTES	Binary data

Development

Setup Development Environment

git clone https://github.com/airbook/airbook-duckdb-utils.git
cd airbook-duckdb-utils
pip install -e .[dev]

Running Tests

pytest tests/

Code Formatting

black airbook_duckdb_utils/
flake8 airbook_duckdb_utils/
mypy airbook_duckdb_utils/

Publishing to Private PyPI

Build the Package

python setup.py sdist bdist_wheel

Upload to Private PyPI

twine upload --repository-url https://your-private-pypi-server.com/legacy/ dist/*

Authentication

For private PyPI access, configure your ~/.pypirc:

[distutils]
index-servers = 
    private-pypi

[private-pypi]
repository: https://your-private-pypi-server.com/legacy/
username: your-username
password: your-password

Or use environment variables:

export TWINE_USERNAME=your-username
export TWINE_PASSWORD=your-password
export TWINE_REPOSITORY_URL=https://your-private-pypi-server.com/legacy/

License

MIT License - see LICENSE file for details.

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Run the test suite
6. Submit a pull request

Changelog

v1.0.0

• Initial release
• DuckDB to BigQuery type mapping functionality
• Support for all major DuckDB data types
• Comprehensive test coverage