Philosophy

There are many aspects to the philosophy, but the crux of it is based on the data hierarchy. The hierarchy currently consists of three levels: component, unit and group levels. Usually, the data unit (the middle level) represents one simulation/experimental run, but it can be defined in any way that suites the problem. The data unit consists of data components (the lowest level), and those represent different parts of your unit like images, waveforms, or tables, for example. The data group consists of multiple data units. Usually, the data group represents simulation/experimental sweeps. The D-Manage approach allows users to focus on processing the data unit and components, and dmanage automatically can deploy the methods to the entire data group efficiently.

Other aspects of the philosophy focus on best practices and tools/components to aid in processing data efficiently, like parallel processing, caching data, metadata, visualization, and remote data access.

Basic Project Procedure

Data Visualization

Eventually data generation and data analysis will feed into each other for efficient data generation and visualization. This offloads the data organization tasks from the user so they can focus on analysis.

Basic Use

Follow the tutorials in the documentation. Use the template to create your first DataUnit and DataGroups.

Remote Computing

The dmanage package utilizes Pyro5 to implement remote procedure calls (RPC) to interact with servers. If your data lives on a remote server, this is the way to go! Basically, the user’s data objects can be generated on the server, and the client can directly access those objects through a proxy and receive the results locally. Of course, the network speed limits how much data that can be transferred practically, but often times results are much more compact than the raw data itself, so this works well!

This functionality is not provided by default, but is easy to enable. Install a fork of the Pyro5 project to your environment using the following command:

pip install dmanage[Pyro5]

Anaconda

create an environment, install pip, and then install the editable package:

conda create -n dmanage
conda activate dmanage
conda install pip
pip install -e <path/to/dmanage>

to see the environment in Spyder, v6 requires spyder-kernels==3.0. PyQt5 is included to enable qt interactive plots:

conda activate dmanage
pip install spyder-kernels==3.1.* PyQt5

Create Conda Package

First, create the PyPi package following the instructions . Then use grayskull to make the conda equivalent yaml file:

pip install grayskull
conda install -c conda-forge conda-recipe-manager
grayskull pypi dmanage