ciocore 5.4.3

Core functionality for Conductor's client tools

Conductor Core

Use the command-line uploader and downloader or develop your own tools using the Python API.

Install

To install the latest version.

pip install --upgrade ciocore --target=$HOME/Conductor

To install a specific version, for example 0.1.0.

pip install --upgrade --force-reinstall ciocore==0.1.0 --target=$HOME/Conductor

Run the conductor command to confirm the package was installed.

$HOME/ConductorStaging/bin/conductor --help

To use the Python API, you must either set the PYTHONPATH or pip install, possibly to a virtualenv.

Option 1. PYTHONPATH

export PYTHONPATH=${HOME}/Conductor && python

Option 2. Pip install to venv.

# Create.
python3.9 -m venv ~/venvs/cioapi

# activate
. ~/venvs/cioapi/bin/activate

pip install --upgrade .

Example API usage from.

from ciocore import api_client

print api_client.request_software_packages()[0]

Contributing

Clone the repo.

git clone [e-mail]:ConductorTechnologies/ciocore.git
cd ciocore

Set up a clean virtual envirionment for development (optional).

python -m virtualenv venv
. ./venv/bin/activate

Install development dependencies

pip install -r requirements_dev.txt

Run the automated tests

tox

Build and install from this directory into to a convenient location for testing.

pip install --upgrade . --target=$HOME/ConductorStaging

See above for information on the conductor command and the Python API.

License

MIT

Changelog

Version:5.4.3 -- 10 Oct 2022

• Add support for customer-supplied KMS encryption keys. [a98c705]

Version:5.4.2 -- 10 Aug 2022

• Adds CONDUCTOR_AUTH_PORT environment variable. [eeee8c5]

Version:5.4.1 -- 27 Jul 2022

• Graft old uploader back in. [caedcce]

Version:5.3.2 -- 25 Jul 2022

• Fix argparse bug https://bugs.python.org/issue16308 [27841c8]

Version:5.3.1 -- 28 Jun 2022

• Adds a check if any errors occured before marking the upload as complete. [124bbf6]

Version:5.3.0 -- 03 Jun 2022

• Adds the option to upload a list of explicit paths to the cli. [ba16fe4]

Version:5.2.2 -- 23 May 2022

• A minor documentation change. [533c7ee]

Version:5.2.0 -- 26 Apr 2022

• Reinstate frozen requirement on urllib version, since it broke submissions on Mac.
• Adds several doc strings.
• Core data now allows initialization with a list of host products to act as a whitelist. Legacy behavior still works.
• Remove reload code.
• Use manageable fixtures for tests.
• Refactor package tree to make docs clearer.
• Autodoc setup, including script to publish autogen doc to mkdocs.

Version:5.1.1 -- 05 Apr 2022

• Ensures the base64 strings are represented as strings. [be95b83]

Version:5.1.0 -- 05 Apr 2022

• Adds registering the uploader and downloader for the user agent. [18a0a7c]
• Adds the user-agent http header to all requests. [6b452c5]

Version:5.0.1 -- 31 Mar 2022

• In developer mode we add X drive letter if submitting to a windows target. [bc91524]
• Adds Packagetree platform filtering.
• Add all=true to the r4equest for packages in order to get Windows packages included. [2735781]
• ciocore.data singleton ensures that no windows -packages are provided if there are no windows instance types, and vice versa.
• Remove urllib constraint because pip complained. Might have to add this back if we get weird breakages. [4e34359]

Version:5.0.0-rc.1 -- 18 Mar 2022

• This version contains breaking changes. Clients should be upgraded.
• PackageTree now supports platform suffixes.
• The singleton that brings together inst_types and packages now filter's each automatically based on platforms provided by the other.
• Improvements to PackageEnvironment, including a constraint to prevent a merge_policy change, and platform detection for correct path separator.
• Better validation for the value of the CONDUCTOR_URL variable.
• Several uploader improvements, including support for reporting progress to external threads and gracefull cancelling the upload process.

Version:4.0.7 -- 10 Mar 2022

• Adds a flag to propogate log properties. [dc12f84]

Version:4.0.6 -- 03 Mar 2022

• Removes automatic setup of logging. adds an option to disable console logging during setup. [9194b4b]

Version:3.4.1 -- 03 Mar 2022

• Adds a centralized py2/3 compatible reload logic. [1f5db2a]
• Avoid validators being duplicated

Version:3.3.0 -- 27 Jan 2022

• Package tree now supports platform suffixes, such as windows or linux. If a platform was provided from the packages request, it will be reflected in the composed package name. [9cb3566]

Version:3.2.4 -- 27 Jan 2022

• Corrects very poor downloader performance in Python3 when runing in daemon mode.

Version:3.2.3 -- 13 Jan 2022

• Repaired broken uploader args resolution. Specifically, md5 caching was None by default, which was treated as OFF. [615bfbe]

Version:3.1.0 -- 16 Nov 2021

• Adds a post_install script to reset the shebang. This avoids errors that could occur when the Python installation in Companion is changed or removed. [3918011]
• Conductor cmd-line now asks for python 3. [093d9ea]

Version:3.0.11 -- 21 Oct 2021

• Rename original downloader to downloader and remove linux daemon downloader version. [6b54566]
• Use a fallback folder for downloads if the output folder doesn't exist. Also support a comma separated task id list. [fb40e8d]
• conductor command finds python on Windows in a cleaner way now. [0ab3261]

Version:3.0.10 -- 01 Sep 2021

• The Windows bat command is simplified. It just tries to use python.exe in the path, and if it fails it provides some help. [0ab3261]

Version:3.0.9 -- 27 Aug 2021

• Fixes a bug where DCCs on Windows using Python2 would fail to upload because the result of a file stat size was a long, and long was erroneousl removed from the type check while converting to python 3 compatibility.

Version:3.0.7 -- 03 Aug 2021

• Fix urllib version because of an error when using proxies. See https://stackoverflow.com/a/66669839/179412 [b89593e]
• Keep auth_url for backwards compat. [1bfe5ed]
• Eliminate use of project_url such as atomic-light-001. [b687f7a]

Version:3.0.6 -- 20 Jul 2021

• Fixed a bug where filenames and md5s were failing assertions in py2 based hosts, like C4DR21 and Maya2019. [0ad6a88]

Version:3.0.5 -- 19 Jul 2021

• Requests exceptions attrs changed, so we now use a try except block for both eventualities. [5f198c8]
• Keep requests under 2.25.1 for m19 because chardet is replaced in 2.26.0 with a buggy replacement charset_normalizer. [f5ebadb]

Version:3.0.4 -- 17 Jul 2021

• Fixed bug where server.py global vars were not reset at start of an auth run(). This caused a problem when credentials were manually deleted during a single python session, which could happen if we suggest that course of action in a support session. [6b4132c]

Version:3.0.3 -- 12 Jul 2021

• Threading Isalive function was renamed to is_alive in py 3.9, so we now try both. [0070709]
• Disable multiprocessing cached bearer token in order to work around a bug in c4d24, where the app itself would respawn recursively if our plugin was registered. [27afab9]
• Use any python version to run the conductor command. Shebang looks for python, not python2. [fd13011]
• Fix bug in pkgs and add windows path support. [fb7a71e]

Version:3.0.2 -- 27 Jun 2021

• Remove the index.html file in favour of an inline string. [05fab98]

Version:3.0.1 -- 20 Jun 2021

• Removes dependency on builtins which caused an error when running through Python2.7 [a37b118]
• Better spawning for conductor command on Windows. It now searches the path for Python versions with a preference for Python3 [ed11102]
• Config is no longer a singleton instantiated on initialization. It can be force refreshed, although that functionality is not yet utilized. [1407582]

Version:2.0.1 -- 10 Jun 2021

• Adds Python 3 support for VFX Platform (CY2020) Maya2022, C4d23 etc. [fa5005b]

Version:1.0.3 -- 10 Jun 2021

• Restored the please-close-your-browser index.html page that appears after authentication. [ba11cb1]

Version:1.0.2 -- 29 Apr 2021

• Fix logging - use const for logger name. [38111bf]
• Adds get_api_key_from_variable function. [cd57428]

Version:1.0.0 -- 07 Apr 2021

• Version 1.0.0 is not backward compatible with older versions. The functionality from sequence, gpath, and expander has been extracted to separate Py3 compatible packages. [25c64c6]

Version:0.3.0 -- 23 Mar 2021

• Gpath supports dollar brace variables for context expansion. [ff2fae9]

Version:0.2.26 -- 11 Mar 2021

• Better local dev flow and instructions. [4566a0b]

Version:0.2.25 -- 11 Mar 2021

• Adds skulk to deps. [4c2dc89]
• Developer install instructions.. [b9afd1d]

Version:0.2.20 -- 11 Mar 2021

• Set up repo for Circle CI deploy. [e111736]

Version:0.2.19 -- 24 Feb 2021

• Adds basename modifiers to expander. Example, use <basename scenefile> instead of <scenename> [daf1af3]

Version:0.2.18 -- 15 Jan 2021

• Adds expression resolution to expander class. [cbeb64e]
• Unc support (#16)

Version:0.2.16 -- 07 Jan 2021

• Gpath supports UNC paths. [7333433]

Version:0.2.15 -- 06 Nov 2020

• Developer can switch between using test-fixtures and the real API within one session by using data.set_fixtures_dir() method. As a result, the DISK_CACHE env var is no longer used and has been deleted. [226b860]

Version:0.2.14 -- 21 Oct 2020

• Repair windows command line executable. Added a bat file so that it is no longer necessary to call python explicitly. [cf9f19f]

Version:0.2.13 -- 08 Oct 2020

• Adds README and CHANGELOG to PyPi description. [97a07dd]

Version:0.2.12 -- 04 Oct 2020

• Catch unimplemented thread_count. [c64ad79]

Version:0.2.11 -- 01 Oct 2020

• Package environment now allows exclusive override. [481851b]
• Allow relative paths that start with letter, colon - it could happen. [241fc68]

Version:0.2.10 -- 08 Sep 2020

• Gpath allow colons in paths. [9c19d3a]

Version:0.2.9 -- 06 Sep 2020

• Sequence consumers must use factory. [01a7dd7]

Version:0.2.8 -- 03 Sep 2020

• Tidy and remove dependency on future. [0c2f72c]

Version:0.2.7 -- 28 Aug 2020

• Adds cycle_progressions chunk strategy for improved scout frame distribution. [9b42288]
• Remove yaml config (#8)
• Remove config file references
• Adds jwt domain validation for credentials file so token is renewed when switching Google projects.
• Api_url defaults to same domain as auth_url

Version:0.2.6 -- 18 Aug 2020

• Adds jwt domain validation for creds file. [0042b2b]
• Remove config file references. [451896b]

Version:0.2.5 -- 11 Aug 2020

• Production release

Version:0.2.4 -- 11 Aug 2020

• Removed redshift package ids hack. [8d11367]

Version:0.2.3 -- 10 Aug 2020

• Adds remove_missing files method to Gpath. [3b96796]
• Adds validators base class. [c91ffc9] [b074cce]

Version:0.2.2 -- 08 Aug 2020

• Temp workarounds for missing plugin-host links in packages
• Remove "shared" request.session object. (#293)
• Update httpbatchworker docstring (#291)
• File api refactor multipart and tcp connection pooling (#290)
• Multipart updates to workers
• Add make_preapred_request for s3 calls to remove transfer-encoding header being added. update descriptions and v2 endpoints. Add content-length for s3 calls. don't return response object on s3 calls which can cause a build up of memory due to jobworker, return headers or none.
• Add metric_store increments for aws presigned and multipart, cannot use chunked reader since generator does not have len function.
• Use httpbatchworker response to avoid additional os.stat calls. [7c76c46]

Version:0.2.0 -- 01 Aug 2020

• Use explicit entry point script. [dfd9da3]
• The coredata singleton that holds instance_types, projects, and software must now be initialized with software product. This is to avoid having to specify the product on every call to data(). [db0fcb0]
• Expander context made public to indicate to other objects that they may retrieve the context. [a457aa1]

Version:0.1.15 -- 27 Jul 2020

• Made post install script runnable. [8dce89f]

Version:0.1.14 -- 27 Jul 2020

• Simplified post install script - no longer interactive. [6bba8af]

Version:0.1.13 -- 26 Jul 2020

• Refactored post install setup script. [3fbba5e]

Version:0.1.12 -- 25 Jul 2020

• Setup can accept cmdline args for non-wizard mode. [f195b7d]
• Adds safe mode to expander. [95f6cfd]

Version:0.1.11 -- 22 Jul 2020

• Adds Setup wizard to make installation easier. [a06d610]

Version:0.1.10 -- 21 Jul 2020

• Use cio to differentiate from conductor. [3f49451]
• Projects and instance types sorted for fixtures. [4a0f833]

Version:0.1.9 -- 10 Jul 2020

• Implement data singletons here and remove from conductor-maya. [d31aeaa]

Version:0.1.8 -- 06 Jul 2020

• Repair some path list issues found while adding remove method. [7368cc2]

Version:0.1.7 -- 04 Jul 2020

• Adds a remove method to pathlist. [5712d2c]
• Wip configure script. [e7f6ea9]

Version:0.1.6 -- 29 Jun 2020

• Fix bad path to conductor cmd. [5be12af]
• Packages: Remove tree property and make _tree -> tree. [1e6ae34]

Version:0.1.5 -- 22 Jun 2020

• Remove src. [a366047]
• Newline in requirements. [a4cd9d3]
• Missing member. [b2bfb6d]
• Fix tests. [c9498c9]
• Imports ciocore. [916651e]
• Flatten. [c88afae]
• Flatter hierarchy. [6906e23]
• Slug means local slug. [0db16e0]
• Init declare namespace. [84d16b3]
• Use underscore name. [1aae967]
• Correct namespace config. [22a09f3]

Version:0.1.4 -- 14 Jun 2020

• Ignore build dir. [8080df5]