oddrn-generator 0.1.29

Open Data Discovery Resource Name Generator

Open Data Discovery Resource Name Generator

Requirements

• Python >= 3.7

Installation

poetry add oddrn-generator
# or
pip install oddrn-generator

Usage and configuration

Available generators

• cassandra - CassandraGenerator
• postgresql - PostgresqlGenerator
• mysql - MysqlGenerator
• glue - GlueGenerator
• s3 - S3Generator
• kafka - KafkaGenerator
• kafkaconnect - KafkaConnectGenerator
• snowflake - SnowflakeGenerator
• airflow - AirflowGenerator
• hive - HiveGenerator
• dynamodb - DynamodbGenerator
• odbc - OdbcGenerator
• mssql - MssqlGenerator
• oracle - OracleGenerator
• redshift - RedshiftGenerator
• clickhouse - ClickHouseGenerator
• athena - AthenaGenerator
• quicksight - QuicksightGenerator
• dbt - DbtGenerator
• prefect - PrefectGenerator
• tableau - TableauGenerator
• neo4j - Neo4jGenerator

Work in progress generators

• kubeflow - KubeflowGenerator
• dvc - DVCGenerator
• great_expectations - GreatExpectationsGenerator

Generator properties

• base_oddrn - Get base oddrn (without path)
• available_paths - Get all available path of generator

Generator methods

• get_oddrn_by_path(path_name, new_value=None) - Get oddrn string by path. You also can set value for this path using 'new_value' param
• set_oddrn_paths(**kwargs) - Set or update values of oddrn path
• get_data_source_oddrn() - Get data source oddrn

Generator parameters:

• host_settings: str - optional. Hostname configuration
• cloud_settings: dict - optional. Cloud configuration
• **kwargs - path's name and values

Example usage

# postgresql
from oddrn_generator import PostgresqlGenerator
oddrn_gen = PostgresqlGenerator(
  host_settings='my.host.com:5432', 
  schemas='schema_name', databases='database_name', tables='table_name'
)

oddrn_gen.base_oddrn
# //postgresql/host/my.host.com:5432
oddrn_gen.available_paths
# ('schemas', 'databases', 'tables', 'columns')

oddrn_gen.get_data_source_oddrn()
# //postgresql/host/my.host.com:5432/schemas/schema_name/databases/database_name

oddrn_gen.get_oddrn_by_path("schemas")
# //postgresql/host/my.host.com:5432/schemas/schema_name

oddrn_gen.get_oddrn_by_path("databases")
# //postgresql/host/my.host.com:5432/schemas/schema_name/databases/database_name

oddrn_gen.get_oddrn_by_path("tables")
# //postgresql/host/my.host.com:5432/schemas/schema_name/databases/database_name/tables/table_name

# you can set or change path:
oddrn_gen.set_oddrn_paths(tables='another_table_name', columns='new_column_name')
oddrn_gen.get_oddrn_by_path("columns")
# //postgresql/host/my.host.com:5432/schemas/schema_name/databases/database_name/tables/another_table_name/columns/new_column_name

# you can get path wih new values:
oddrn_gen.get_oddrn_by_path("columns", new_value="another_new_column_name")
# //postgresql/host/my.host.com:5432/schemas/schema_name/databases/database_name/tables/another_table_name/columns/another_new_column_name


# glue
from oddrn_generator import GlueGenerator
oddrn_gen = GlueGenerator(
  cloud_settings={'account': 'acc_id', 'region':'reg_id'}, 
  databases='database_name', tables='table_name', columns='column_name', 
  jobs='job_name', runs='run_name', owners='owner_name'
)

oddrn_gen.available_paths
# ('databases', 'tables', 'columns', 'owners', 'jobs', 'runs')

oddrn_gen.get_oddrn_by_path("databases")
# //glue/cloud/aws/account/acc_id/region/reg_id/databases/database_name

oddrn_gen.get_oddrn_by_path("tables")
# //glue/cloud/aws/account/acc_id/region/reg_id/databases/database_name/tables/table_name'

oddrn_gen.get_oddrn_by_path("columns")
# //glue/cloud/aws/account/acc_id/region/reg_id/databases/database_name/tables/table_name/columns/column_name

oddrn_gen.get_oddrn_by_path("jobs")
# //glue/cloud/aws/account/acc_id/region/reg_id/jobs/job_name

oddrn_gen.get_oddrn_by_path("runs")
# //glue/cloud/aws/account/acc_id/region/reg_id/jobs/job_name/runs/run_name

oddrn_gen.get_oddrn_by_path("owners")
# //glue/cloud/aws/account/acc_id/region/reg_id/owners/owner_name

Exceptions

• WrongPathOrderException - raises when trying set path that depends on another path

from oddrn_generator import PostgresqlGenerator
oddrn_gen = PostgresqlGenerator(
    host_settings='my.host.com:5432', 
    schemas='schema_name', databases='database_name',
    columns='column_without_table'
)
# WrongPathOrderException: 'columns' can not be without 'tables' attribute

• EmptyPathValueException - raises when trying to get a path that is not set up

from oddrn_generator import PostgresqlGenerator
oddrn_gen = PostgresqlGenerator(
    host_settings='my.host.com:5432', schemas='schema_name', databases='database_name',
)
oddrn_gen.get_oddrn_by_path("tables")

# EmptyPathValueException: Path 'tables' is not set up

• PathDoestExistException - raises when trying to get not existing oddrn path

from oddrn_generator import PostgresqlGenerator
oddrn_gen = PostgresqlGenerator(
    host_settings='my.host.com:5432', schemas='schema_name', databases='database_name',
)
oddrn_gen.get_oddrn_by_path("jobs")

# PathDoestExistException: Path 'jobs' doesn't exist in generator

Development

#Install dependencies
poetry install

#Activate shell
poetry shell

# Run tests
python run pytest