ODD Collector SDK

Root project for ODD collectors

Domain

CollectorConfig

Main config file for collector

class CollectorConfig(pydantic.BaseSettings):
    default_pulling_interval: int # pulling interval in minutes
    token: str                    # token for requests to odd-platform
    plugins: Any
    platform_host_url: str

Collector

Args:

config_path: str - path to collector_config.yaml (i.e. '/collector_config.yaml')

root_package: str - root package for adapters which will be loaded (i.e. 'my_collector.adapters')

plugins_union_type - Type variable for pydantic model.

Plugin

Is a config for adapter

class Plugin(pydantic.BaseSettings):
  name: str
  description: Optional[str] = None
  namespace: Optional[str] = None

Plugin class inherited from Pydantic's BaseSetting,it means it can take any field, which was skipped in collector_config.yaml, from env variables.

Field type: Literal["custom_adapter"] is obligatory for each plugin, by convention literal MUST have same name with adapter package

Plugins example:

  # plugins.py
  class AwsPlugin(Plugin):
      aws_secret_access_key: str
      aws_access_key_id: str
      aws_region: str
  
  class S3Plugin(AwsPlugin):
      type: Literal["s3"]
      buckets: Optional[List[str]] = []

  class GluePlugin(AwsPlugin):
      type: Literal["glue"]
  
  # For Collector's plugins_union_type argument
  AvailablePlugin = Annotated[
      Union[
          GluePlugin,
          S3Plugin,
      ],
      pydantic.Field(discriminator="type"),
  ]

AbstractAdapter Abstract adapter which MUST be implemented by generic adapters

Collector example

Requirenments

Use the package manager poetry to install add odd-collector-sdk and asyncio.

poetry add odd-collector-sdk

A typical top-level collector's directory layout (as an example we took poetry project)

.
├── my_collector            
│   ├── adapters            # Adapters
│   │   ├── custom_adapter  # Some adapter package
│   │   │   ├── adapter.py  # Entry file for adapter
│   │   │   └── __init__.py
│   │   ├── other_custom_adapter
│   │   ├── ...             # Other adapters
│   │   └── __init__.py
│   ├── domain              # Domain models
│   │   ├── ...
│   │   ├── plugins.py      # Models for available plugins
│   │   └── __init__.py
│   ├── __init__.py         
│   └── __main__.py         # Entry file for collector
├── ...
├── collector_config.yaml
├── pyproject.toml
├── LICENSE
└── README.md

Adapters folder

Each adapter inside adapters folder must have an adapter.py file with an Adapter class implementing AbstractAdapter

    # custom_adapter/adapter.py example
    from odd_collector_sdk.domain.adapter import AbstractAdapter
    from odd_models.models import DataEntityList

    # 
    class Adapter(AbstractAdapter):
        def __init__(self, config: any) -> None:
            super().__init__()

        def get_data_entity_list(self) -> DataEntityList:
            return DataEntityList(data_source_oddrn="test")

        def get_data_source_oddrn(self) -> str:
            return "oddrn"

Plugins

Each plugin must implement Plugin class from sdk

    # domain/plugins.py
    from typing import Literal, Union
    from typing_extensions import Annotated

    import pydantic
    from odd_collector_sdk.domain.plugin import Plugin

    class CustomPlugin(Plugin):
        type: Literal["custom_adapter"]


    class OtherCustomPlugin(Plugin):
        type: Literal["other_custom_adapter"]

    # Needs this type variable for Collector initialization
    AvailablePlugins = Annotated[
        Union[CustomPlugin, OtherCustomPlugin],
        pydantic.Field(discriminator="type"),
    ]

collector_config.yaml

default_pulling_interval: 10 
token: "" 
platform_host_url: "http://localhost:8080" 
plugins:
  - type: custom_adapter
    name: custom_adapter_name
  - type: other_custom_adapter
    name: other_custom_adapter_name

Usage

# __main__.py

import asyncio
import logging
from os import path


from odd_collector_sdk.collector import Collector

# Union type of avalable plugins
from my_collector.domain.plugins import AvailablePlugins

logging.basicConfig(
    level=logging.INFO, format="[%(asctime)s] %(levelname)s in %(module)s: %(message)s"
)

try:
    cur_dirname = path.dirname(path.realpath(__file__))
    config_path = path.join(cur_dirname, "../collector_config.yaml")
    root_package = "my_collector.adapters"

    loop = asyncio.get_event_loop()

    collector = Collector(config_path, root_package, AvailablePlugin)

    loop.run_until_complete(collector.register_data_sources())

    collector.start_polling()
    loop.run_forever()
except Exception as e:
    logging.error(e, exc_info=True)
    loop.stop()

And run

poetry run python -m my_collector

Metadata

Release files for odd-collector-sdk 0.3.60

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for odd-collector-sdk 0.3.60
File	Size	Uploaded	Provenance
odd_collector_sdk-0.3.60.tar.gz	34.0 kB	Nov 6, 2024	Without provenance	Details

Built distribution (wheel)

Table of built distributions (wheels) for odd-collector-sdk 0.3.60
File	Interpreter	ABI	Platform	Reset
odd_collector_sdk-0.3.60-py3-none-any.whl 32.4 kB Nov 6, 2024 Without provenance	Python 3	none	any	Details

Total release size: 66.4 kB

Release files / odd_collector_sdk-0.3.60.tar.gz

Download URL	odd_collector_sdk-0.3.60.tar.gz
Size	34.0 kB
Tags	Source
SHA-256 checksum How to use checksums	`4e57756863f9f08c61be82c216c030fb727807accbdd0b16575a34e16eae1c70`
BLAKE2b-256 checksum How to use checksums	`ea0b2f0c6d080ad28fe938d226973f3a929b9f69ad71f7ea5ea28505c486f870`
Upload date	Nov 6, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`poetry/1.3.2 CPython/3.9.16 Linux/6.5.0-1025-azure`

Release files / odd_collector_sdk-0.3.60-py3-none-any.whl

Download URL	odd_collector_sdk-0.3.60-py3-none-any.whl
Size	32.4 kB
Tags	Python 3
SHA-256 checksum How to use checksums	`0773bf6e51d9c0569ff34a8b560ce9dea2d9ff2877e7b4b2e8f87df75419e993`
BLAKE2b-256 checksum How to use checksums	`d563d368a3621641aa664a541b84c2408de50b2fff630ef16f3e789b9853d85a`
Upload date	Nov 6, 2024
Uploaded using Trusted Publishing? What is trusted publishing?	No
Uploaded via	`poetry/1.3.2 CPython/3.9.16 Linux/6.5.0-1025-azure`

Security for odd-collector-sdk 0.3.60

Report project as malware

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.