Methods to help track the scripts and datafiles in a project.
Datatracker is a basic logging Python package that keeps track of files and code within a Project. Each script is logged as an entry and input and output datafiles are recorded in order. Datatracker is able to manage versioning of both files and scripts, and is able to identify the most up-to-date version.
At the moment, this Python package is still in alpha, and I may include changes to both UI and file format that may be breaking.
To install, run the following command:
pip install git+ssh://email@example.com/TarjinderSingh/datatracker
For an entry,
tagis a unique identifier to the script in question and should be clear what the general purpose and output of the script is. (ie Merge is not what we want to see here)
descriptionneeds to be one or two sentences equivalent of the Git commit message that thoroughly describes the general purpose and output of the script.
categoryindicates the general step of analysis the script belongs to.
moduleis the sub-category for which the script belongs to. Type
category_templatein interactive Python for an idea of the appropriate categories and modules are.
For a InputFile or OutputFile,
tagis a unique identifier to the File in question and should be clear what the general purpose and output of the script is. (ie Merge is not what we want to see here).
descriptionfor a file is a one or two sentences equivalent of the Git commit message that thoroughly describe the general purposes of the File at hand.
from datatracker import * tr = Tracker() os.environ['VERSION'] = '0.1.0' entry = Entry(tag='filter-common-variants', description='Filtering common variants in new GWAS data set.', category='Processing', module='Variant QC') infile = entry.add( InputFile(tag='raw-plink-file', path='gs://bucket/raw-plink-file.bed', description='Raw PLINK file.')) outfile = entry.add( OutputFile(tag='filt-plink-file', path='gs://bucket/raw-plink-file.bed', description='Filtered PLINK file.')) tr.save(entry)
View existing entries
from datatracker import * tr = Tracker() tr.table
Use existing entries for pipeline
infile = entry.add(InputFile(entry_tag='filter-common-variants', tag='raw-plink-file', database=tr))
Filter and remove
# filter to entry tr.filter(tr.entry.tag_version == 'import-array_0.1.6') # remove entry tr.remove(tr.entry.tag_version == 'import-array_0.1.6')
Pandas and Excel
df = tr.explode() df = tr.explode('filt-plink-file') df = tr.to_pandas() df = tr.table df.to_excel('spreadsheet.xlsx')
infile = entry.add(InputFile(path='gs://checkpoint-cache/tmp/1.bed'))
MIT License (see repository)
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Hashes for datatracker-0.2.5-py3-none-any.whl