Data Warehouse Client
This package provides access to the e-Science Central data warehouse that can be used to store, access and analyse data collected in scientific studies, including for healthcare applications. The primary aim of the warehouse was to create a general system that enables users to explore data collected in a variety of forms. This might include data collected through questionnaires, data collected from sensors, and features extracted from the analysis of sensor data (e.g. activity levels derived from raw accelerometer data). Researchers might wish to slice, dice, visualise, analyse and explore this data in different ways, e.g. all results for one participant, all results for one type of measure in a study, changes in measurements over time. Others may wish to build models that can then be used in applications that make predictions about future values.
Traditionally, data collected in studies has been stored in a collection of files,
often with metadata encoded in the filenames.
This makes it difficult, and time consuming, for researchers to explore, interpret and analyse the data.
The data warehouse exploits modern database technology to vastly simplify this effort.
In doing this we have drawn heavily on the best practice for data warehouse design.
However, there is more variety in the types of healthcare data to be stored than there is in a typical warehouse,
and so we have been forced to deviate from a conventional data warehouse in some aspect of the design.
There are three guiding principles behind the design:
- The data warehouse must be able to store any type of data collected in a study without modifying the schema. This means that when new types of data are collected in studies (e.g. from a new questionnaire, a new data analysis program, or a new sensor) they can be stored in the warehouse without any changes to its design. This has 3 main advantages: firstly, it enables us to fix and optimise the schema for the tables in which the data is stored; secondly it means that applications and tools (e.g. for analysis and visualisation) built on the warehouse do not have to be updated when new types of data are added; thirdly, a single, multi-tenant database server can support many studies. This reduces the overall costs, the start-up time for a new study, and the overheads of managing the warehouse.
- Descriptive information about the types of measurement is stored in the warehouse so that tools or humans can interpret the data stored there.
- The design is optimised for query performance. In several cases, this has led to denormalization (duplication of data) to reduce the need for expensive joins.
- It must support a security regime to restrict each user’s access to the data collected in studies.
For more information see: P. Watson and H. Hiden, "The e-Science Central Study Data Platform" 2022 IEEE 18th International Conference on e-Science (e-Science), Salt Lake City, UT, USA, 2022, pp. 55-64, doi: 10.1109/eScience55777.2022.00020. https://scholar.google.co.uk/citations?view_op=view_citation&hl=en&user=KQJg3lwAAAAJ&sortby=pubdate&citation_for_view=KQJg3lwAAAAJ:z0_F5_TITjQC
For more documentation see A Data Warehouse for Storing and Analysing Study Data.
Running Instructions
To install from PyPi, run:
pip install data-warehouse-client
In directory in which your executable is run, create a db-credentials.json file containing database
credentials (substituting all <VARS>):
{"user": "<USER>", "pass": "<PASSWORD>", "IP": "<IP>", "port": <PORT>}
1 maintainer
Metadata
1 maintainer
Release files for data-warehouse-client 3.0.7
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| data_warehouse_client-3.0.7.tar.gz | 807.3 kB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| data_warehouse_client-3.0.7-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 887.4 kB
Release files / data_warehouse_client-3.0.7.tar.gz
| Download URL | data_warehouse_client-3.0.7.tar.gz |
|---|---|
| Size | 807.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
92e29c762d4c9410fa20a6890011ce11f4ded6ff3ddc74a882f016ee939a0096
|
|
BLAKE2b-256 checksum How to use checksums |
3994576c13bfdb09def9c4bbdef0a1f2c1704ba888ff04b48e8d94a88040324d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/5.1.1 CPython/3.12.5
|
Release files / data_warehouse_client-3.0.7-py3-none-any.whl
| Download URL | data_warehouse_client-3.0.7-py3-none-any.whl |
|---|---|
| Size | 80.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
b3af3f30e15ba6ff96f167db78cd013ef5841d05978e0516da33debd84a40591
|
|
BLAKE2b-256 checksum How to use checksums |
3d35be04f1943697afd25aefdbc62eaaa7a0beea904d28a3b88fb68162317a5b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/5.1.1 CPython/3.12.5
|
Security for data-warehouse-client 3.0.7
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
