Python package for extracting files from IFS to CDF. Copyright 2020 Cognite AS
Project description
CDF IFS file extractor
In progress
General
The cdf-ifs-file-extractor allows continuous uploading, changing and deletion of files stored in a local directory and connect those files to the corresponding assets with some metadata. Running on a continuous schedule the extractor moves old mapping_files into an Archive and uploads new files if it is fed with a new mapping file. Feedback of unsuccessful upload attempts is given via the two files unknown_files.csv and unknown_assets.csv.
Structure of the Mapping file
The extractor expects the mapping file to have a specific format. This requires the following columns:
- FILE_NAME
- DOC_NO
- DOC_REV
- DOC_SHEET
- DOC_CLASS
- DOC_CLASS_NAME
- DOC_FORMAT
- DOC_FORMAT_NAME
- DOC_TITLE
- DOC_RESP_SIGN
- LETTER_DATE
- LAST_UPDATED: to be checked if this column is really included
- DOC_REFERENCE_OBJECT
- OPERATION: allows one of the three values N, C, D to upload a new file or change or delete an existing one
Requirements local directory
The local directory needs to contain
- the desired files to operate on
- a folder with the name Archive
- the mapping file
Parameters
The parameters for the function are
- project:
- api_key_name: name of the environment variable storing the api key
- base_url
- client_name
- client_timeout
- log_path
- log_level
- path: path of the local directory
- mapping_file: file name
Feedback of the extractor
For non developing users the extractor provides feedback to the user for unsuccessful file operation attempts in two cases
- The asset_id specified in the mapping file does not exist: these ids are stored together with the corresponding file name in unknown_assets.csv
- The specified file does not exist: unknown_files.csv
Both files will be created in the end of a run of the extractor and archived as soon as the extractor is running the next time.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for cdf-ifs-file-extractor-0.1.2.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | b85797b165ff7bf5a65a6f35d8d78dd0ff6fae1d510648bbf16bd77a0963a69f |
|
MD5 | f5481102d8c1fb3129f13a093fbc8d98 |
|
BLAKE2b-256 | f14f3b5c9eb99974a47622da0e06a9126bc2933455112eb61d99f229caf9d4d2 |
Hashes for cdf_ifs_file_extractor-0.1.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 60a8223cc5786353b6c869f81f23508d667cac05bdfe278385d3e3c5aef686ea |
|
MD5 | 8035ae45b3038bb6dcd4e9fe6c08e7f3 |
|
BLAKE2b-256 | 7b4270f1ddf26a58b85105ba267a49e001a86417b9fea3ea5d645c576f200646 |