A package to simplify connecting to the TM1 REST API from Apache Airflow
Project description
airflow-providers-tm1
A package by Knowledgeseed and Cubewise that provides a hook to simplify connecting to the IBM Cognos TM1 / Planning Analytics REST API.
https://github.com/KnowledgeSeed/airflow-providers-tm1
This repository is a fork of https://github.com/MariusWirtz/airflow-tm1 and https://github.com/scrambldchannel/airflow-tm1 which offer only Airflow 1.x compatibility.
The fork upgrades the provider to ensure Airflow 2.x compatibility. Some parts and have been reused from https://github.com/scrambldchannel/airflow-provider-tm1.
Requirements
- Python 3.7+
- Airflow 2.2+
- TM1py 1.4.1+
Development
python -m venv .env
source .env/bin/activate
python -m pip install -r requirements.txt
python -m build
Installation
Install with pip pip install airflow-providers-tm1
Usage
Create a connection in Airflow with at least the following parameters set:
- Host
- Login
- Password
- Port
- Extras
- ssl
Any other parameter accepted by the TM1py RestService constructor (eg base_url, namespace etc) can also be added as a key in the Extras field in the connection.
In your DAG file:
from airflow_providers_tm1.operators.tm1_run_ti import TM1RunTIOperator
...
t1 = TM1RunTIOperator (
task_id='t1',
tm1_conn_id='tm1_conn',
process_name='airflow_test_params_success_dag',
tm1_params={'testParam1': 'testParamValue'},
timeout=20,
cancel_at_timeout=True
)
This will attempt to connect to the TM1 server using the details provided and initialise an instance of the TM1Service class than be accessed at airflow_providers_tm1.hooks.tm1.TM1Hook
See TM1py for more details.
It's important to mention that TM1Py will execute TI process in asynchronous mode. The operator submits the request, receives the async_id
from TM1 and starts polling the result using async_id
until the response is retrieved or times out.
Timeout is defined in seconds, default value is 300 seconds. It's also important that Airflow-side timeout does not
involve automatically the cancellation of the TI process. If cancel_at_timeout
is set True
, Airflow will make an attempt
to cancel the long-running TI process.
For further examples, please see tests_integration/dags
folder.
Manual integration testing
Use tests_integration/docker-compose.yaml
as a baseline, which spins up an Airflow including the TM1 provider and a base TM1 database to test against. Please note that tm1-docker
image is properietary IBM product wrapped in Docker by Knowledgeseed and therefore it is only available internally for Knowledgeseed developers.
To obtain a licensed IBM TM1 database for testing or production purpose, please see https://www.ibm.com/topics/tm1 for further details.
License
See LICENSE
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file airflow_providers_tm1-0.0.10.tar.gz
.
File metadata
- Download URL: airflow_providers_tm1-0.0.10.tar.gz
- Upload date:
- Size: 16.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.0 CPython/3.9.19
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 35592e8d45d5f4cece0fa8493c44aa7ec17af2c05e722413739a97ebabb53111 |
|
MD5 | 61c9f683b3d41fefb0f643be20ac9f02 |
|
BLAKE2b-256 | a836c84b5d74424184216e25c6f0f14c45b0f2bc2157d73285b59ee186ad0268 |
Provenance
File details
Details for the file airflow_providers_tm1-0.0.10-py3-none-any.whl
.
File metadata
- Download URL: airflow_providers_tm1-0.0.10-py3-none-any.whl
- Upload date:
- Size: 15.7 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/5.1.0 CPython/3.9.19
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1fea37957a05b32aeb4463e4a3f6e8624c5e7fed221c97a31ea1a0135c0f597b |
|
MD5 | 3966f4d308a12f66a702d3737b18553e |
|
BLAKE2b-256 | b434b061c00342995a413b1a42a3ea0b4231d2670d7a71e611bfa922643d750d |