Skip to main content

A package to simplify connecting to the TM1 REST API from Apache Airflow

Project description

airflow-providers-tm1

CI Build

A package by Knowledgeseed and Cubewise that provides a hook to simplify connecting to the IBM Cognos TM1 / Planning Analytics REST API.

https://github.com/KnowledgeSeed/airflow-providers-tm1

This repository is a fork of https://github.com/MariusWirtz/airflow-tm1 and https://github.com/scrambldchannel/airflow-tm1 which offer only Airflow 1.x compatibility.

The fork upgrades the provider to ensure Airflow 2.x compatibility. Some parts and have been reused from https://github.com/scrambldchannel/airflow-provider-tm1.

Requirements

  • Python 3.7+
  • Airflow 2.2+
  • TM1py 1.4.1+

Development

python -m venv .env
source .env/bin/activate
python -m pip install -r requirements.txt
python -m build

Installation

Install with pip pip install airflow-providers-tm1

Usage

Create a connection in Airflow with at least the following parameters set:

  • Host
  • Login
  • Password
  • Port
  • Extras
    • ssl

Any other parameter accepted by the TM1py RestService constructor (eg base_url, namespace etc) can also be added as a key in the Extras field in the connection.

airflow_tm1_conn

In your DAG file:

from airflow_providers_tm1.operators.tm1_run_ti import TM1RunTIOperator

...

t1 = TM1RunTIOperator (
        task_id='t1',
        tm1_conn_id='tm1_conn',
        process_name='airflow_test_params_success_dag',
        tm1_params={'testParam1': 'testParamValue'},
        timeout=20,
        cancel_at_timeout=True
    )

This will attempt to connect to the TM1 server using the details provided and initialise an instance of the TM1Service class than be accessed at airflow_providers_tm1.hooks.tm1.TM1Hook

See TM1py for more details.

It's important to mention that TM1Py will execute TI process in asynchronous mode. The operator submits the request, receives the async_id from TM1 and starts polling the result using async_id until the response is retrieved or times out. Timeout is defined in seconds, default value is 300 seconds. It's also important that Airflow-side timeout does not involve automatically the cancellation of the TI process. If cancel_at_timeout is set True, Airflow will make an attempt to cancel the long-running TI process.

For further examples, please see tests_integration/dags folder.

Manual integration testing

Use tests_integration/docker-compose.yaml as a baseline, which spins up an Airflow including the TM1 provider and a base TM1 database to test against. Please note that tm1-docker image is properietary IBM product wrapped in Docker by Knowledgeseed and therefore it is only available internally for Knowledgeseed developers.

To obtain a licensed IBM TM1 database for testing or production purpose, please see https://www.ibm.com/topics/tm1 for further details.

License

See LICENSE

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

airflow_providers_tm1-0.0.10.tar.gz (16.7 kB view details)

Uploaded Source

Built Distribution

airflow_providers_tm1-0.0.10-py3-none-any.whl (15.7 kB view details)

Uploaded Python 3

File details

Details for the file airflow_providers_tm1-0.0.10.tar.gz.

File metadata

File hashes

Hashes for airflow_providers_tm1-0.0.10.tar.gz
Algorithm Hash digest
SHA256 35592e8d45d5f4cece0fa8493c44aa7ec17af2c05e722413739a97ebabb53111
MD5 61c9f683b3d41fefb0f643be20ac9f02
BLAKE2b-256 a836c84b5d74424184216e25c6f0f14c45b0f2bc2157d73285b59ee186ad0268

See more details on using hashes here.

File details

Details for the file airflow_providers_tm1-0.0.10-py3-none-any.whl.

File metadata

File hashes

Hashes for airflow_providers_tm1-0.0.10-py3-none-any.whl
Algorithm Hash digest
SHA256 1fea37957a05b32aeb4463e4a3f6e8624c5e7fed221c97a31ea1a0135c0f597b
MD5 3966f4d308a12f66a702d3737b18553e
BLAKE2b-256 b434b061c00342995a413b1a42a3ea0b4231d2670d7a71e611bfa922643d750d

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page