ts-sdk
Tetrascience Python SDK
Version
v2.7.0
Table of Contents
Install
pip3 install ts-sdk
Task Script
Config.json
Overview
All Task Scripts have a config.json file that describes:
- what
languagea Task Script is written in - the
functionsa Task Script has available for a Protocol to use - the
runnerType, an optional field that is only used when letting the artifact builder know a Windows script (identified by using the value “windows-script”) is being built - the
maxCountof container instances that will be used by the task - and an optional field describing the amount of memory a task will need,
memoryInMB
Allowed IDS
Within the functions object of the configuration, the optional field allowedIds describes the IDS types a Task Script can produce.
The allowedIds field is used by the Context API's context.write_file (when writing an IDS) and context.write_ids functions to validate that the ids parameter of either function is allowed before writing an IDS.
allowedIds can be used in the following ways:
-
A single object with three properties:
namespace,slug,version-
When a function only generates one kind of IDS
-
The
idsparameter of the context.write_file and context.write_ids functions are optional since there is only one value in theallowedIdsfield -
Example:
"functions": [ { "slug": "generates-ids", "function": "main.generate_ids", "allowedIds": { "namespace": "common", "slug": "my-ids", "version": "v1.0.0" } } ]
-
-
An array of those objects with the three properties:
namespace,slug,version-
When a function can generate multiple IDS types
-
The
idsparameter of the context.write_file and context.write_ids functions is required and will be validated against theallowedIdsto confirm the task script is allowed to write that IDS -
Example:
"functions": [ { "slug": "generates-ids", "function": "main.generate_ids", "allowedIds": [ { "namespace": "common", "slug": "my-ids", "version": "v1.0.0" }, { "namespace": "common", "slug": "my-ids", "version": "v1.1.0" }, { "namespace": "common", "slug": "my-other-ids", "version": "v1.0.0" } ] } ]
-
-
null-
When a function does not generate any IDS
-
Calls to context.write_file (when
file_categoryis “IDS”) and context.write_ids will raise an exception- When the
file_categoryparameter is not "IDS", then context.write_file is not writing an IDS and therefore will not raise an exception
- When the
-
Example:
"functions": [ { "slug": "generates-ids", "function": "main.generate_ids", "allowedIds": null } ]
-
-
allowedIdsnot defined- Any IDS JSON can be written via context.write_file or context.write_ids
- no allowed IDSs are specified, so no check is performed
- Any IDS JSON passed to the context.write_file or context.write_ids still has to pass validation against the IDS specified by the
idsparameter
- Any IDS JSON can be written via context.write_file or context.write_ids
Changelog
Show
v2.8.0
- Add a
no_propagateflag tocontext.delete_labels(..) - Added
context.enable_experiment(..)for opting into experimental features- Use
context.enable_experiment("conditional_writes")to use conditional writes oncontext.write_file(..)
- Use
v2.7.0
- Add prerelease support for context.get_ids(...) if pipeline has prereleases enabled
- Add testing support for
context.read_file(form="download")andcontext.read_file(form="file_obj")
v2.6.3
- Fix OOM error handling in Lambda task scripts
testingsubmodule: Files in Task Scripts should always return bytes, not strings
v2.6.2
- Enable passing
Authdirectly toTask - Add basic stub support to
context.run_cmdin testing mode - Add a pattern matching API to
Task().run() - Repair
get_idsandget_schema_artifactin secure mode
v2.6.1
- Fix ts-cfg.json api_url usage to remove trailing "/v1"
context.write_detached_filemissing ids parameter fix
v2.6.0
-
Add support for running Task Scripts locally
example
from ts_sdk.testing.models import * local_file = LocalFile(path='test-file.json') with Trigger(file=local_file) as trigger: result, error = Task(function='main').run({'hello': 'world'}) print(result) print(local_file.labels)
-
Add support for Schema Artifact integration in Task Scripts: introduced
get_schema_artifactand ArtifactUtilDict for handling Schema Artifacts. -
Improve logger to include
tsSdkVersionin each log entry (visible in debug mode).
v2.5.1
- Set
ts_user_ids3 file meta for new file in datalake
v2.5.0
- Adjust publish to support Codebuild build_id in response for all artifact types
v2.4.1
- pass source_name and source_id in
context.write_detached_file(default to pipeline name and id)
v2.4.0
- new
context.write_detached_fileto write workflow input independent file
v2.3.0
- Add
return_commandparameter tocontext.run_cmdandcontext.run_command(default:False) - When set toTrue, returns the entire command response object from TDP instead of just the response body. Default behavior continues to returncommand.responseBodyon success or raise an exception containingcommand.responseBodyon failure.
v2.2.4
- Remove trailing comma from JSON changed in v2.2.2
v2.2.3
- Restore the explicit signing version when generating a pre-signed URL
v2.2.2
- Move
allowedIdsin the task script config.json schema from the Config object to the Function object - Add the new
successWhenandonFailedfields to the protocol v3 json schema
v2.2.1
- Use timeout=300s for internal task endpoints
v2.2.0
- Refactoring to avoid duplications with artifact-builder (for task script lambda feature)
v2.1.3
- Update
context.validate_idsto validate IDS instances usingts-ids-validator:v1.1.0- Check that the instance is valid against the schema
- Check that the shape of
datacubes[*].measures[*].valuesmatchesdatacubes[*].dimensions[*].scale
- Update the cli's deprecation date
v2.1.2
- Add exponential backoff to the task script runner
v2.1.1
- Pass the current workflow id to the file info service when mutating labels
v2.1.0
- Update logic for
context.run_cmdmethod (also applied tocontext.run_command):- Implemented exponential backoff for polling the command service for command status
- Introduced an optional
initial_delay_secparameter for initial delayed polling - Reduced the minimum
ttl_secthreshold from 300 seconds to 60 seconds
- Adds an optional query argument to
context.search_eql
v2.0.2
- Update IDS artifact validation: now when using
ts-sdk putto upload an IDS artifact, breaking change validation runs by downloading and comparing the previous version of the IDS from the Tetra Data Platform - see thets-ids-validatorpackage for more detail.
v2.0.1
- Pin
tenacitydependency to resolve compatibility issue with latest version
v2.0.0
- Secure mode + bug fixes
v1.4.2
- upgrade
smart_open(v4.2.0→v6.3.0)
v1.4.1
- Make
allowedIDSnon-mandatory for task-script, to maintain backward compatibility
v1.4.0
- Added functionality to support
allowedIdsfeature
v1.3.10
- Internal fixes (
/update-statusAPI call retry)
v1.3.9
CLI Improvements
- Make artifact build failure more evident to the user
ts-sdk putwill check forrequirements.txtfile before uploading
Context Function Improvements
- Add support to
write_fileto receive a dictionary instead of only strings - Validate dictionary inputs to
write_fileagainst IDS - Enforce IDS input to
write_idsmust be a dictionary - Validate length of the name and value of labels
- Validate labels more thoroughly, in more places
- Add a new DataClass for better label definition
Other
- All code formatted with
black - Adjust abstract dependencies in
setup.pyto be less strict so that it will work better within task scripts - Replace
jsonlibrary withsimplejsonfor speed and usability
v1.3.8
- Internal fixes (secrets handling)
v1.3.7
- Add
--exclude-foldersargument tots-sdk put task-scriptthat excludes common folders that generally do not need to be part of the uploaded task script (e.g..git,example-input,__tests__) - Add local check to prevent uploading artifacts using
ts-sdk putthat would be rejected by the server for being too large - Improve error messages for adding invalid labels to files
v1.3.6
- Add new s3 meta:
- DO_NOT_INHERIT_LABELS
- CONTENT_CREATED_FROM_FILE_ID
v1.3.5
- Fix bug where datalake file_key was incorrectly generated
v1.3.2
- Update
context.write_file()to validate file upload path - Fix logging issues
- Improve namespace validation
- Update
printfunctionality to be more accurate and group arguments to the same call together
Metadata
Release files for ts-sdk 2.8.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| ts_sdk-2.8.0.tar.gz | 77.4 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| ts_sdk-2.8.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 185.3 kB
Release files / ts_sdk-2.8.0.tar.gz
| Download URL | ts_sdk-2.8.0.tar.gz |
|---|---|
| Size | 77.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1c261f1c158c40d47d6705d841c33cce22235b974c050c5e77c73db99bade0b8
|
|
BLAKE2b-256 checksum How to use checksums |
038033a4b14ed640a8977cc8e10b76956796efa1cc3e2e1917dd6994445377ea
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Release files / ts_sdk-2.8.0-py3-none-any.whl
| Download URL | ts_sdk-2.8.0-py3-none-any.whl |
|---|---|
| Size | 107.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
403bb30634ab025e5792215eb03a6c5d7a0a04f2f63e34b8640d2276a42ad193
|
|
BLAKE2b-256 checksum How to use checksums |
435f8886ffc6e3e42e8cb3e9bc53f140316dd40b27736a8658d91b93ead93fa8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|