Python package for extracting files from IFS to CDF. Copyright 2020 Cognite AS
Project description
CDF IFS file extractor
In progress
General
The cdf-ifs-file-extractor allows continuous uploading, changing and deletion of files stored in a local directory and connect those files to the corresponding assets with some metadata. Running on a continuous schedule the extractor moves old mapping_files into an Archive and uploads new files if it is fed with a new mapping file. Feedback of unsuccessful upload attempts is given via the two files unknown_files.csv and unknown_assets.csv.
Structure of the Mapping file
The extractor expects the mapping file to have a specific format. This requires the following columns:
- FILE_NAME
- DOC_NO
- DOC_REV
- DOC_SHEET
- DOC_CLASS
- DOC_CLASS_NAME
- DOC_FORMAT
- DOC_FORMAT_NAME
- DOC_TITLE
- DOC_RESP_SIGN
- LETTER_DATE
- LAST_UPDATED: to be checked if this column is really included
- DOC_REFERENCE_OBJECT
- OPERATION: allows one of the three values N, C, D to upload a new file or change or delete an existing one
Requirements local directory
The local directory needs to contain
- the desired files to operate on
- a folder with the name Archive
- the mapping file
Parameters
The parameters for the function are
- project:
- api_key_name: name of the environment variable storing the api key
- base_url
- client_name
- client_timeout
- log_path
- log_level
- path: path of the local directory
- mapping_file: file name
Feedback of the extractor
For non developing users the extractor provides feedback to the user for unsuccessful file operation attempts in two cases
- The asset_id specified in the mapping file does not exist: these ids are stored together with the corresponding file name in unknown_assets.csv
- The specified file does not exist: unknown_files.csv
Both files will be created in the end of a run of the extractor and archived as soon as the extractor is running the next time.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for cdf-ifs-file-extractor-0.1.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | f519b6a5fbd18840358f4d7a57468b6bacefe9e2746af778c71ef5e0f4b30ed6 |
|
MD5 | cf4dc71930baf348dbaf9f69897b335b |
|
BLAKE2b-256 | 954fe8cb9847b72f267c224c71b104d266c811d3ab31ab27a88b2437f7facba8 |
Hashes for cdf_ifs_file_extractor-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 4e7a619e15eee01680cff20e44aeb31fb211b512e9534caf793d6c1fc8d6b9f5 |
|
MD5 | 1ff5f3314099bc780742fd9c6f40afbe |
|
BLAKE2b-256 | 401c7674d4ea7c2329825cd565020c2d88d0f347078b4e681b06846d62021f7f |