Secure Infrastructure for Research with Administrative Data
Secure Infrastructure for Research with Administrative Data (SIRAD)
sirad is an integration framework for data from administrative systems. It
deidentifies administrative data by removing and replacing personally
identifiable information (PII) with a global anonymized identifier, allowing
researchers to securely join data on an individual from multiple tables without
knowing the individual's identity. It is developed by
Research Improving People's Lives (RIPL).
For a worked example and further details, please see sirad-example.
To learn more about the motivation for creating this package and its potential uses, please see our article in Communications of the ACM:
J.S. Hastings, M. Howison, T. Lawless, J. Ucles, P. White. (2019). Unlocking Data to Improve Public Policy. Communications of the ACM 62(10): 48-53. doi:10.1145/3335150
Requires Python 3.7 or later.
To install from PyPI using pip:
pip install sirad
To install using Anaconda Python:
conda install -c ripl-org sirad
To install a development version from the current directory:
pip install -e .
There is a single command line script included,
sirad supports the following arguments:
process- split raw data files into data and PII files
research- create a versioned set of research files with a unique anonymous identifier
To set configuration options, create a file called
sirad_config.py and place
either in the directory where you are executing the
sirad command or
somewhere else on your Python path. See
config.py for a
complete list of possible options and default values.
The following options are available:
DATA_SALT: secret salt used for hashing data values. This shouldn't be shared. A warning will be outputted if it is not set. Defaults to None.
PII_SALT: secret salt used for hashing pii values. This shouldn't be shared. A warning will be issued if it is not set. Defaults to None.
LAYOUTS: directory that contains layout files. Defaults to
RESEARCH_DIR: paths to where the original data, the processed files, and the research files will be saved.
VERSION: the current version number of the processed and research files.
sirad uses YAML files to define the layout, or structure, of raw data files.
These YAML files define each column in the incoming data and how it should be
processed. More documentation to come on this YAML format.
The following file formats are supported:
- csv - change delimiter with delimiter option
- fixed with
- xlsx (xls not currently supported)
Sample test data is randomly generated using Faker; none of the information identifies real individuals.
- tax.txt - sample tax return data. Includes first, last, DOB and SSN.
- credit_scores.txt - sample credit score information. Includes first, last and DOB but no SSN.
Run unit tests as:
python -m unittest discover
sirad can also be used as an API from custom Python code. Documentation to come.
- Mark Howison
- Ted Lawless
- John Ucles
- Preston White
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.