Flexible Data Integrator
Project description
Flexible Dataset Integrator (FDI) helps data producers and processors to build connections into isolated heterogeneous datasets. to assemble, organize, and integrate data into self-describing, modular, hierarchical, persistent, referenceable Products, whose component datasets keep their own characteristicss and are easily accessible.
FDI provides scripting-friendly APIs and tools to define custom Products and generating Python class files. An integrated Product takes care of inter-platform compatibility, string representation, serialisation to simple exchange format, persistence to disk or server, and carrying references of other Products, enabling context-building and lazy-loading.
FDI’s base data model is defined in sub-package dataset. Persistent data access using storage pools, referencing, and Universal Resource Names, and referencie-carrying product Context, are defined in sub-package pal. A reference RESTful API server designed to communicate with a data processing server/docker using the data model, and a reference HTTP pool server are in sub-package pns.
Install/Uninstall
<a href=”http://mercury.bao.ac.cn:9006/mh/fdi/-/commits/master”><img alt=”pipeline status” src=”http://mercury.bao.ac.cn:9006/mh/fdi/badges/master/pipeline.svg” /></a>
For Users
If you need to use fdi, install from PyPI:
python3 -m pip install fdi
or from git repo:
python3 -m pip install http://mercury.bao.ac.cn:9006/mh/fdi/-/archive/master/fdi-master.tar.gz
If you want to install the develop branch with dependencies:
python3 -m pip install fdi[DEV]
or
python3 -m pip install http://mercury.bao.ac.cn:9006/mh/fdi/-/archive/develop/fdi-develop.tar.gz#egg=fdi[DEV]
To uninstall:
python3 -m pip uninstall fdi
For Developers and Admins (or Those who are Not Sure which to Choose)
To install
FDIINSTDIR=/tmp # change this to your installation dir
cd $FDIINSTDIR
git clone ssh://git@mercury.bao.ac.cn:9005/mh/fdi.git
cd fdi
git checkout develop
make install
If you want to install the master branch, remove the git checkout develop line above.
To test your installation
make test
To Generate Python Product Source Code
Re-generate baseproduct.py and product.py from YAML schema files in fdi/dataset/resources:
make py
Modify/Generate Documents
If you plan to compile documents in the docs directory, generate diagrams, API files, or HTML pages, run (in that order, respectively):
make docs_plots
make docs_api
make docs_html
The generated HTML page is at docs/html/index.html.
Run Servers
If you plan to run the pns and/or the http pool server locally, install the dependencies:
python3 -m pip install -e .[SERV]
make installpns
To test your pns servers installation, in one window, run:
make runserver
in another window run:
make testpns
To test your httppool servers installation, in one window, run:
make runpoolserver
in another window run:
make testhttppool
For More
For more examples see tests/test_*.py.
Read more on package introduction, description, quick start, and API documents on readthedocs.io.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.