Unofficial Treasure Workflow API client
Project description
Unofficial Treasure Workflow API client.
Installation
pip install tdworkflow
If you want to use development version, run as follows:
pip install git+https://github.com/chezou/tdworkflow.git
Usage
import os
from tdworkflow.client import Client
apikey = os.getenv("TD_API_KEY")
client = Client(site="us", apikey=apikey)
# Or, write endpoint explicitly
# client = Client(endpoint="api-workflow.treasuredata.com", apikey=apikey)
projects = client.projects("pandas-df")
secrets = {"td.apikey": apikey, "td.apiserver": "https://api.treasuredata.com", "test": "secret-foo"}
client.set_secrets(projects[0], secrets)
client.secrets(projects[0])
# ['td.apikey', 'td.apiserver', "test"]
client.delete_secrets(projects[0], ["test", "td.apiserver"])
Upload Project from GitHub
Before executing the example code, you have to install git-python
pip install gitpython
Clone example repository with git-python and upload a digdag project.
import tempfile
import os
import shutil
import tdworkflow
from git import Git
# Download example GitHub repositoory
tempdir = tempfile.gettempdir()
git_repo = "https://github.com/treasure-data/treasure-boxes/"
shutil.rmtree(os.path.join(tempdir, "treasure-boxes"))
try:
Git(tempdir).clone(git_repo)
print("Clone repository succeeded")
except Exception:
print("Repository clone failed")
raise
# Upload specific Workflow project
apikey = os.getenv("TD_API_KEY")
site = "us"
target_box = os.path.join("integration-box", "python")
target_path = os.path.join(tempdir, "treasure-boxes", target_box)
client = tdworkflow.client.Client(site=site, apikey=apikey)
project = client.create_project("my-project", target_path)
If you want to open Treasure Workflow console on your browser, you can get the workflow URL as the following:
CONSOLE_URL = {
"us": "https://console.treasuredata.com/app/workflows",
"eu01": "https://console.eu01.treasuredata.com/app/workflows",
"jp": "https://console.treasuredata.co.jp/app/workflows",
}
workflows = client.project_workflows(project)
workflows = list(filter(lambda w: w.name != "test", workflows))
if workflows:
print(f"Project created! Open {CONSOLE_URL[site]}/{workflows[0].id}/info on your browser and click 'New Run' button.")
else:
print("Project creation failed.")
Start workflow session
You can start a workflow session by using Client.start_attempt.
attempt = client.start_attempt(workflows[0])
# Wait attempt until finish. This may require few minutes.
attempt = client.wait_attempt(attempt)
Connect to open source digdag
Since Treasure Workflow is hosted digdag, tdworkflow is compatible with open source digdag.
Here is the example code to connect local digdag server.
>>> import tdworkflow
>>> import requests
>>> session = requests.Session()
>>> client = tdworkflow.client.Client(
... endpoint="localhost:65432", apikey="", _session=session, scheme="http")
>>> client.projects()
[Project(id=1, name='python-tdworkflow', revision='134fe2f9-ded3-4e7c-af8e-8a82d55d688b', archiveType='db', archiveMd5='5Lc6F6m3DtmBN4DA5MzK8A==', createdAt='2019-11-01T13:03:26Z', deletedAt=None, updatedAt='2019-11-01T13:03:26Z')]
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
tdworkflow-0.9.0.tar.gz
(34.4 kB
view hashes)
Built Distribution
tdworkflow-0.9.0-py3-none-any.whl
(35.4 kB
view hashes)
Close
Hashes for tdworkflow-0.9.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6991e1a72f62060d1ce9b6cc03ed30d6143a58ac8869c733491d01d4d9579e71 |
|
MD5 | 52b2706eaedae86fde613b04cb71bcef |
|
BLAKE2b-256 | 1ff8f6ca0046343e8b2a1a13342867ef770d4d1d18da1e3aaa65ca7f7936fbdb |