OpenDataDiscovery Models package
Has some useful tools for working with OpenDataDiscovery. Such as:
- Generated Python models from OpenDataDiscovery specification.
- API Client for working with OpenDataDiscovery.
- API for manual discovering data entities.
Installation
pip install odd-models
Models using example
odd-models.models package provides automatically generated Python model by OpenDataDiscovery specification. It can be used for creating data entities for uploading them into the Platform.
Code example (full code):
from oddrn_generator import PostgresqlGenerator
from odd_models.models import DataEntity, DataSet, DataSetField, DataSetFieldType, DataEntityType, Type, MetadataExtension
generator = PostgresqlGenerator(host_settings="localhost", databases="my_database", schemas="public")
DataEntity(
oddrn=generator.get_oddrn_by_path("tables", "my_table"),
name="my_table",
type=DataEntityType.TABLE,
metadata=[MetadataExtension(schema_url="https://example.com/schema.json", metadata={"env": "DEV"})],
dataset=DataSet(
field_list=[
DataSetField(
oddrn=generator.get_oddrn_by_path("tables_columns", "name"),
name="name",
type=DataSetFieldType(
type=Type.TYPE_STRING,
logical_type='str',
is_nullable=False
),
)
]
)
)
HTTP Client for OpenDataDiscovery
odd-models.client package provides API client for OpenDataDiscovery API. Client provides an API for working with OpenDataDiscovery Platform. It has various methods for working with data sources, data entities, management etc.
Code example(full code):
from examples.postgres_models import data_entity_list, generator
from odd_models.api_client.v2.odd_api_client import Client
client = Client(host="http://localhost:8080")
client.auth(name="postgres", description="Token for dev AWS account data sources")
client.create_data_source(
data_source_oddrn=generator.get_data_source_oddrn(),
data_source_name="Postgres data source",
)
client.ingest_data_entity_list(data_entities=data_entity_list)
Manual Discovery API
When there is no programmatic way to discover data sources and data entities, odd-models.discovery package provides API for manual discovery of data sources and data entities.
Code example(full code):
from odd_models.discovery import DataSource
from odd_models.discovery.data_assets import AWSLambda, S3Artifact
from odd_models.discovery.data_assets.data_asset_list import DataAssetsList
with DataSource("//cloud/aws/dev") as data_source:
validation_lambda = AWSLambda.from_params(
region="eu-central-1", account="0123456789", function_name="validation"
)
input_artifact = S3Artifact.from_url("s3://bucket/folder/test_data.csv")
results = S3Artifact.from_url("s3://bucket/folder/test_result.csv")
metrics = S3Artifact.from_url("s3://bucket/folder/test_metrics.json")
input_artifact >> validation_lambda >> DataAssetsList([results, metrics])
data_source.add_data_asset(validation_lambda)
Development
Installation
# Install dependencies
poetry install
# Activate virtual environment
poetry shell
Generating models
# Generate models. Will generate models pydantic into odd_models/models
make generate_models
# Generate api client. Will generate api client into odd_models/api_client
make generate_client
Tests
pytest .
Docker build
docker build -t odd-models .
Metadata
Release files for odd-models 2.0.51
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| odd_models-2.0.51.tar.gz | 18.5 kB | Without provenance | Details for odd_models-2.0.51.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| odd_models-2.0.51-py3-none-any.whl | Python 3 | none | any | Details for odd_models-2.0.51-py3-none-any.whl |
Total release size: 46.4 kB
Release files / odd_models-2.0.51.tar.gz
| Download URL | odd_models-2.0.51.tar.gz |
|---|---|
| Size | 18.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c2829f7e94c59c853cde1d4796785ae863a35a194606c8d6f67c0bff955e236d
|
|
BLAKE2b-256 checksum How to use checksums |
c22a3bf61fc186838a8c95e526cb92d0791ea0bff63bca2fc6f58a4142ec2c51
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.3.1 CPython/3.9.16 Linux/6.5.0-1025-azure
|
Release files / odd_models-2.0.51-py3-none-any.whl
| Download URL | odd_models-2.0.51-py3-none-any.whl |
|---|---|
| Size | 27.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ac9c036ef16deae8d80bba4ae754bd527b4110589d9bdd087e1224098663a960
|
|
BLAKE2b-256 checksum How to use checksums |
d30e93a52db85d313136860b4362c5eb60e72d763304a54ef12cefe080ce6a2a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.3.1 CPython/3.9.16 Linux/6.5.0-1025-azure
|
Security for odd-models 2.0.51
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.