Official Cognite Data Fusion tool for project templates and configuration deployment
Project description
Official Cognite Data Fusion project templates
The CDF Toolkit is a command-line interface (cdf-tk
) used for configuring and administrating Cognite Data
Fusion (CDF) projects.
It supports three different modes of operation:
- As an interactive command-line tool used alongside the Cognite Data Fusion web application to retrieve and push configuration of the different Cognite Data Fusion services like data sets, data models, transformations, and more. This mode also supports configuration of new Cognite Data Fusion projects to quickly get started.
- As tool to support the project life-cyle by scripting and automating configuration and management of Cognite Data Fusion projects where CDF configurations are kept as yaml-files that can be checked into version control. This mode also supports DevOps workflows with development, staging, and production projects.
- As a tool to deploy official Cognite project templates to your Cognite Data Fusion project. The tool comes bundled with templates useful for getting started with Cognite Data Fusion, as well as for specific use cases delivered by Cognite or its partners. You can also create your own templates and share them.
ALPHA NOTE
The templates and the cdf-tk
tool are currently in ALPHA. The scope in alpha is on Asset
Performance Management focused on Infield (Digital Operator Rounds). The templates and tooling
will be continously improved throughout moving towards beta and general availability.
Below is an overview of the scope of what can be governed through using these templates:
Quickstart
To install the cdf-tk
tool, you need a working Python installation >=3.9 (recommended 3.11).
Run: pip install cognite-toolkit
The cdf-tk
tool is available as a command line tool. Run cdf-tk --help
to see the available commands.
Naming standards
A common structure on naming different CDF resource types and configuration are important from day one. Easy to use and understandable naming standard makes it easy to navigate and search on all components in a project as is grows in data sources, code, configuration, supported solutions and human resources working with and using the CDF solutions.
Separation tokens
- For external IDs the separation token is ’_’ (underscore) - this token works for all OS, when external ID matches usage of files
- For names the separation token is ’:’ or '_' (colon or underscore) - these tokens matches usage in other CLI tools ang gives good readability
Example usage of naming standard
In the example below we are setting up a project based on the Open Industry Data (OID), that originates from the Valhall oil rig. Hence the example location below is oid
- the location_name = oid
- The different data sources are:
- workmate (asset & workorder data)
- fileshare (files and 3D)
- PI (time series / data points)
CDF project
│
├── Data Sets:
│ ├── extId: ds_asset_oid ── name: asset:oid
│ │ ├── Extraction Pipelines:
│ │ │ └── extId: ep_src_asset_oid_workmate ── name: src:asset:oid:workmate
│ │ │
│ │ ├── RAW DB/tables:
│ │ │ └── DB: asset_oid_workmate ── table: assets
│ │ │
│ │ ├── Transformations:
│ │ │ └── extId: tr_asset_oid_workmate_asset_hierarchy ── name: asset:oid:workmate:asset_hierarchy
│ │ │
│ │ └── Autorisation groups:
│ │ ├── id: asset:oid:extractor
│ │ ├── id: asset:oid:prosessing
│ │ └── id: asset:oid:read
│ │
│ ├── extId: ds_files_oid ── name: files:oid
│ │ ├── Extraction Pipelines:
│ │ │ ├── extId: ep_src_files_oid_fileshare ── name: src:files:oid:fileshare
│ │ │ └── extId: ep_ctx_files_oid_fileshare:annotation ── name: ctx:files:oid:fileshare:annotation
│ │ │
│ │ ├── RAW DB/tables:
│ │ │ └── DB: files_oid_fileshare ── table: file_metadata
│ │ │
│ │ ├── Transformations:
│ │ │ └── extId: tr_file_oid_fileshare_file_metadata ── name: file:oid:metadata:fileshare:file_metadata
│ │ │
│ │ ├── Functions:
│ │ │ └── extId: fu_files_oid_fileshare_annotation ── name: files:oid:fileshare:annotation
│ │ │
│ │ └── Autorisation groups:
│ │ ├── id: files:oid:extractor
│ │ ├── id: files:oid:prosessing
│ │ └── id: files:oid:read
│ │
│ ├── extId: ds_workorder_oid ── name: workorder:oid
│ │ ├── ...
│ │ ...
│ │
│ ├── extId: ds_timeseries_oid ── name: timeseries:oid
│ │ ├── ...
│ │ ...
│ │
│ ├── extId: ds_3d_oid ── name: 3d:oid
│ │ ├── ...
│ │ ...
│
└── Spaces:
└── extId: sp_apm_oid ── name: oid
Naming elements
- Data Type: asset, timeseries, workorder, files, 3d,... (use what is relevant for project)
- Source: Source system where data originates from (ex, SAP, Workmate, Aveva, PI, Fileshare, SharePoint,..)
- Location: Location for Asset / System / Plant / installation
- Pipeline Type: src = source data, ctx = contextualization, uc = use case, ...
- Operation Type: Type of operation/action/functionality in transformation or CDF function
- Access Type: Type of access used in authorization groups (ex: extractor, processing, read, ...)
Data sets:
External ID: ds_<data type>_<location>
Name: <data type>:<location>
Ex: ds_asset_oid / asset:oid
Extraction Pipelines:
External ID: ep_<pipeline type>_<data type>_<location>_<source>
Name: <pipeline type>:<data type>:<location>:<source>:
Ex: ep_src_asset_oid_workmate / src:asset:oid:workmate
RAW DB/tables:
DB: <data type>_<location>_<source>
Ex: asset_oid_workmate
Table: use name from source, or other describing name
Transformations:
External ID: tr_<data type>_<location>_<source>_<operation type>
Name: <data type>:<location>:<source>:<operation type>
Ex: tr_asset_oid_workmate_asset_hierarchy / asset:oid:workmate:asset_hierarchy
Functions:
External ID: fu_<data type>_<location>_<source>_<operation type>
Name: <data type>:<location>:<source>:<operation type>
Ex: fu_files_oid_fileshare_annotation / files:oid:fileshare:annotation
Authorization groups:
Name: <data type>:<source>:<access type>
Ex: asset:valhall:extractor / asset:valhall:processing / asset:valhall:read
Data Model Spaces:
External ID: dm_<data type>_<source>
Name: <data type>:<source>
Ex: dm_apm_oid / apm:oid
For more information
More details about the tool can be found at developer.cognite.com.
You can find an overview of the modules and packages in the module and package documentation.
See ./CONTRIBUTING.md for information about how to contribute to the cdf-tk
tool or
templates.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for cognite_toolkit-0.1.0a3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bcc2888884f027c8a44b3a56be5d628eed42f8468eba73c07351537b7900f319 |
|
MD5 | 230cf774ac27c97b865bd1a780e9d491 |
|
BLAKE2b-256 | d467096e85cc9c43da9f63cf20f858e5591ead49307d1a61f2ded10d557927e7 |