Skip to main content

Python package for extracting files from IFS to CDF. Copyright 2020 Cognite AS

Project description

Cognite logo

CDF IFS file extractor

build Code style: black

In progress

General

The cdf-ifs-file-extractor allows continuous uploading, changing and deletion of files stored in a local directory and connect those files to the corresponding assets with some metadata. Running on a continuous schedule the extractor moves old mapping_files into an Archive and uploads new files if it is fed with a new mapping file. Feedback of unsuccessful upload attempts is given via the two files unknown_files.csv and unknown_assets.csv.

Structure of the Mapping file

The extractor expects the mapping file to have a specific format. This requires the following columns:

  • FILE_NAME
  • DOC_NO
  • DOC_REV
  • DOC_SHEET
  • DOC_CLASS
  • DOC_CLASS_NAME
  • DOC_FORMAT
  • DOC_FORMAT_NAME
  • DOC_TITLE
  • DOC_RESP_SIGN
  • LETTER_DATE
  • LAST_UPDATED: to be checked if this column is really included
  • DOC_REFERENCE_OBJECT
  • OPERATION: allows one of the three values N, C, D to upload a new file or change or delete an existing one

Requirements local directory

The local directory needs to contain

  • the desired files to operate on
  • a folder with the name Archive
  • the mapping file

Parameters

The parameters for the function are

  • project:
  • api_key_name: name of the environment variable storing the api key
  • base_url
  • client_name
  • client_timeout
  • log_path
  • log_level
  • path: path of the local directory
  • mapping_file: file name

Feedback of the extractor

For non developing users the extractor provides feedback to the user for unsuccessful file operation attempts in two cases

  • The asset_id specified in the mapping file does not exist: these ids are stored together with the corresponding file name in unknown_assets.csv
  • The specified file does not exist: unknown_files.csv

Both files will be created in the end of a run of the extractor and archived as soon as the extractor is running the next time.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for cdf-ifs-file-extractor, version 0.1.1
Filename, size File type Python version Upload date Hashes
Filename, size cdf_ifs_file_extractor-0.1.1-py3-none-any.whl (16.0 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size cdf-ifs-file-extractor-0.1.1.tar.gz (8.5 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page