Python package for extracting files from IFS to CDF. Copyright 2020 Cognite AS
Project description
CDF IFS file extractor
In progress
General
The cdf-ifs-file-extractor allows continuous uploading, changing and deletion of files stored in a local directory and connect those files to the corresponding assets with some metadata. Running on a continuous schedule the extractor moves old mapping_files into an Archive and uploads new files if it is fed with a new mapping file. Feedback of unsuccessful upload attempts is given via the two files unknown_files.csv and unknown_assets.csv.
Structure of the Mapping file
The extractor expects the mapping file to have a specific format. This requires the following columns:
- FILE_NAME
- DOC_NO
- DOC_REV
- DOC_SHEET
- DOC_CLASS
- DOC_CLASS_NAME
- DOC_FORMAT
- DOC_FORMAT_NAME
- DOC_TITLE
- DOC_RESP_SIGN
- LETTER_DATE
- LAST_UPDATED: to be checked if this column is really included
- DOC_REFERENCE_OBJECT
- OPERATION: allows one of the three values N, C, D to upload a new file or change or delete an existing one
Requirements local directory
The local directory needs to contain
- the desired files to operate on
- a folder with the name Archive
- the mapping file
Parameters
The parameters for the function are
- project:
- api_key_name: name of the environment variable storing the api key
- base_url
- client_name
- client_timeout
- log_path
- log_level
- path: path of the local directory
- mapping_file: file name
Feedback of the extractor
For non developing users the extractor provides feedback to the user for unsuccessful file operation attempts in two cases
- The asset_id specified in the mapping file does not exist: these ids are stored together with the corresponding file name in unknown_assets.csv
- The specified file does not exist: unknown_files.csv
Both files will be created in the end of a run of the extractor and archived as soon as the extractor is running the next time.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for cdf-ifs-file-extractor-0.1.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | efed728d9b93ff71d55ec4c198417a843d42f5d3303a71559f39d712f015bcec |
|
MD5 | 17f8762f15e0beedbfe923e8dc40e7b2 |
|
BLAKE2b-256 | 92444f0e3d515e24552e535f4b57f9a6a5e614ec1ce298fa8dfaf420805d78cd |
Hashes for cdf_ifs_file_extractor-0.1.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | bb366f14c110cb4a2a4155c3e9e13725a69b91d8a62521a0118d823481b14fae |
|
MD5 | 1bb862d602550f8ae8d068503fc38c6f |
|
BLAKE2b-256 | 6758adc0d2f668f2d097b087f501a02eee12f27ee1589e0e50f832479cb16f2f |