Open source offering of the cdisc rules engine
Project description
Supported python versions
cdisc-rules-engine
Open source offering of the cdisc rules engine
Quick start
To quickly get up and running with CORE, users can download the latest executable version of the engine for their operating system from here: https://github.com/cdisc-org/cdisc-rules-engine/releases
Once downloaded, simply unzip the file and run the following command based on your Operating System:
Windows:
.\core.exe validate -s <standard> -v <standard_version> -d path/to/datasets
# ex: .\core.exe validate -s sdtmig -v 3-4 -d .\xpt\
Linux/Mac:
./core validate -s <standard> -v <standard_version> -d path/to/datasets
# ex: ./core validate -s sdtmig -v 3-4 -d .\xpt\
Code formatter
This project uses the black
code formatter and flake8
linter for python.
It also uses pre-commit
to run black
and flake8
when you commit.
Both dependencies are added to requirements.txt.
Required
Setting up pre-commit
requires one extra step. After installing it you have to run
pre-commit install
This installs pre-commit
in your .git/hooks
directory.
Installing dependencies.
These steps should be run before running any tests or core commands using the non compiled version.
- Create a virtual environment:
python -m venv <virtual_environment_name>
- Activate the virtual environment:
./<virtual_environment_name>/bin/activate
-- on linux/mac
.\<virtual_environment_name>\Scripts\Activate
-- on windows
- Install the requirements.
python -m pip install -r requirements.txt
# From the root directory
Running The Tests
From the root of the project run the following command:
python -m pytest tests/unit/
Running a validation
From the command line
Clone the repository and run python core.py --help
to see the full list of commands.
Run python core.py validate --help
to see the list of validation options.
-ca, --cache TEXT Relative path to cache files containing pre
loaded metadata and rules
-ps, --pool-size INTEGER Number of parallel processes for validation
-d, --data TEXT Path to directory containing data files
-dp, --dataset-path TEXT Absolute path to dataset file. Can be specified multiple times.
-l, --log-level [info|debug|error|critical|disabled|warn]
Sets log level for engine logs, logs are
disabled by default
-rt, --report-template TEXT File path of report template to use for
excel output
-s, --standard TEXT CDISC standard to validate against
[required]
-v, --version TEXT Standard version to validate against
[required]
-ct, --controlled-terminology-package TEXT
Controlled terminology package to validate
against, can provide more than one
-o, --output TEXT Report output file destination
-of, --output-format [JSON|XLSX]
Output file format
-rr, --raw-report Report in a raw format as it is generated by
the engine. This flag must be used only with
--output-format JSON.
-dv, --define-version TEXT Define-XML version used for validation
--whodrug TEXT Path to directory with WHODrug dictionary
files
--meddra TEXT Path to directory with MedDRA dictionary
files
-r, --rules TEXT Rule core id. ex: CORE-000001. Can be specified multiple times.
-vo, --verbose-output Specify this option to print rules as they
are completed
-p, --progress [verbose_output|disabled|percents|bar]
Defines how to display the validation
progress. By default a progress bar like
"[████████████████████████████--------]
78%"is printed.
--help Show this message and exit.
Available log levels
debug
- Display all logsinfo
- Display info, warnings, and error logswarn
- Display warnings and errorserror
- Display only error logscritical
- Display critical logs
Validate folder
To validate a folder using rules for SDTM-IG version 3.4 use the following command:
python core.py validate -s sdtmig -v 3-4 -d path/to/datasets
Understanding the Rules Report
The rules report tab displays the run status of each rule selected for validation
The possible rule run statuses are:
SUCCESS
- The rule ran and data was validated against the rule. May or may not produce resultsSKIPPED
- The rule was unable to be run. Usually due to missing required data, but could also be cause by rule execution errors.
Additional Core Commands
-
update-cache - update locally stored cache data (Requires an environment variable -
CDISC_LIBRARY_API_KEY
)python core.py update-cache
To obtain an api key, please follow the instructions found here: https://wiki.cdisc.org/display/LIBSUPRT/Getting+Started%3A+Access+to+CDISC+Library+API+using+API+Key+Authentication. Please note it can take up to an hour after sign up to have an api key issued
-
list-rules - list rules available in the cache
- list all rules:
python core.py list-rules
- list rules for standard:
python core.py list-rules -s sdtmig -v 3-4
-
list-rule-sets - lists all standards and versions for which rules are available:
python core.py list-rule-sets
-
test - Test authored rule given dataset in json format
-ca, --cache TEXT Relative path to cache files containing pre
loaded metadata and rules
-dp, --dataset-path TEXT Absolute path to dataset file
-s, --standard TEXT CDISC standard to validate against
[required]
-v, --version TEXT Standard version to validate against
[required]
-ct, --controlled-terminology-package TEXT
Controlled terminology package to validate
against, can provide more than one
-dv, --define-version TEXT Define-XML version used for validation
--whodrug TEXT Path to directory with WHODrug dictionary
files
--meddra TEXT Path to directory with MedDRA dictionary
files
-r, --rule TEXT Path to rule json file.
--help Show this message and exit.
EX: python core.py test -s sdtmig -v 3-4 -dp <path to dataset json file> -r <path to rule json file> --meddra ./meddra/ --whodrug ./whodrug/
Note: JSON dataset should match the format provided by the rule editor:
{
"datasets": [{
"filename": "cm.xpt",
"label": "Concomitant/Concurrent medications",
"domain": "CM",
"variables": [
{
"name": "STUDYID",
"label": "Study Identifier",
"type": "Char",
"length": 10
}
],
"records": {
"STUDYID": [
"CDISC-TEST",
"CDISC-TEST",
"CDISC-TEST",
"CDISC-TEST"
],
}
}
]
}
PyPI Quickstart: Validate data within python
An alternative to running the validation from the command line is to instead import the rules engine library in python and run rules against data directly (without needing your data to be in .xpt
format).
Step 0: Install the library
pip install cdisc-rules-engine
In addition to installing the library, you'll also want to download the rules cache (found in the resources/cache
folder of this repository) and store them somewhere in your project.
Step 1: Load the Rules
The rules can be loaded into an in-memory cache by doing the following:
import os
import pathlib
from multiprocessing.managers import SyncManager
from cdisc_rules_engine.services.cache import InMemoryCacheService
class CacheManager(SyncManager):
pass
# If you're working from a terminal you may need to
# use SyncManager directly rather than define CacheManager
CacheManager.register("InMemoryCacheService", InMemoryCacheService)
def load_rules_cache(path_to_rules_cache):
cache_path = pathlib.Path(path_to_rules_cache)
manager = CacheManager()
manager.start()
cache = manager.InMemoryCacheService()
files = next(os.walk(cache_path), (None, None, []))[2]
for fname in files:
with open(cache_path / fname, "rb") as f:
cache.add_all(pickle.load(f))
return cache
Rules in this cache can be accessed by standard and version using the get_rules_cache_key
function.
from cdisc_rules_engine.utilities.utils import get_rules_cache_key
cache = load_rules_cache()
# Note that the standard version is separated by a dash, not a period
cache_key_prefix = get_rules_cache_key("sdtmig", "3-4")
rules = cache.get_all_by_prefix(cache_key_prefix)
rules
will now be a list of dictionaries the following keys
core_id
- e.g. "CORE-000252"
domains
- e.g.
{'Include': ['DM'], 'Exclude': []}
or{'Include': ['ALL']}
- e.g.
author
reference
sensitivity
executability
description
authorities
standards
classes
rule_type
conditions
actions
datasets
output_variables
Step 2: Prepare your data
In order to pass your data through the rules engine, it must be a pandas dataframe of an SDTM dataset. For example:
>>> data
STUDYID DOMAIN USUBJID AESEQ AESER AETERM ... AESDTH AESLIFE AESHOSP
0 AE 001 0 Y Headache ... N N N
[1 rows x 19 columns]
Before passing this into the rules engine, we need to wrap it in a DatasetVariable.
from cdisc_rules_engine.models.dataset_variable import DatasetVariable
dataset = DatasetVariable(data)
Step 3: Run the (relevant) rules
Next, we need to actually run the rules. We can select which rules we want to run based on the domain of the data we're checking and the "Include"
and "Exclude"
domains of the rule.
# Get the rules for the domain AE
# (Note: we're ignoring ALL domain rules here)
ae_rules = [
rule for rule in rules
if "AE" in rule["domains"].get("Include", [])
]
There's one last thing we need before we can actually run the rule, and that's a COREActions
object. This object will handle generating error messages should the rule fail.
To instantiate a COREActions
object, we need to pass in the following:
results
: An array to which errors will be appendedvariable
: Our DatasetVariabledomain
: e.g. "AE"rule
: Our rule
from cdisc_rules_engine.models.actions import COREActions
rule = ae_rules[0]
results = []
core_actions = COREActions(
results,
variable=dataset,
domain="AE",
rule=rule
)
All that's left is to run the rule!
from business_rules.engine import run
was_triggered = run(
rule=rule,
defined_variables=dataset_variable,
defined_actions=core_actions,
)
Step 5: Interpret the results
The return value of run will tell us if the rule was triggered.
- A
False
value means that there were no errors - A
True
value means that there were errors
If there were errors, they will have been appended to the results array passed into your COREActions
instance. Here's an example error:
{
'executionStatus': 'success',
'domain': 'AE',
'variables': ['AESLIFE'],
'message': 'AESLIFE is completed, but not equal to "N" or "Y"',
'errors': [
{'value': {'AESLIFE': 'Maybe'}, 'row': 1}
]
}
Creating an executable version
Linux
pyinstaller core.py --add-data=venv/lib/python3.9/site-packages/xmlschema/schemas:xmlschema/schemas --add-data=resources/cache:resources/cache --add-data=resources/templates:resources/templates
Windows
pyinstaller core.py --add-data=".venv/Lib/site-packages/xmlschema/schemas;xmlschema/schemas" --add-data="resources/cache;resources/cache" --add-data="resources/templates;resources/templates"
Note .venv should be replaced with path to python installation or virtual environment
This will create an executable version in the dist
folder. The version does not require having Python installed and
can be launched by running core
script with all necessary CLI arguments.
Creating .whl file
All non-python files should be listed in MANIFEST.in
to be included in the distribution.
Files must be in python package.
Unix/MacOS
python3 -m pip install --upgrade build
python3 -m build
To install from dist folder
pip3 install {path_to_file}/cdisc_rules_engine-{version}-py3-none-any.whl
To upload built distributive to pypi
python3 -m pip install --upgrade twine
python3 -m twine upload --repository {repository_name} dist/*
Windows(Untested)
py -m pip install --upgrade build
py -m build
To install from dist folder
pip install {path_to_file}/cdisc_rules_engine-{version}-py3-none-any.whl
To upload built distributive to pypi
py -m pip install --upgrade twine
py -m twine upload --repository {repository_name} dist/*
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for cdisc_rules_engine-0.6.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8138581136df23bfcf192ebdfffe2de3114130376442cf746f8051d5e1e1a7f9 |
|
MD5 | 97112a67fc04285a45c775dd47dc54d2 |
|
BLAKE2b-256 | a16363481dcca5b2abbe1249266ae335270830554a6a2eb664e1d44219286a06 |