rsbids 0.0.1a1

No project description provided

rsbids

rsbids is a rust implementation of pybids, currently under active development. It offers vastly improved runtimes compared to other bids indexers (benchmarks to come), a streamlined core api, and a pybids compatibility api.

rsbids is currently in alpha. Most of the core pybids features are implemented, however, there is little to no automated testing or documentation. It has only rudimentary validation and no configurability. Pybids compatibility has been implemented for much of pybids.layout.layout, pybids.layout.indexers, and pybids.layout.models. Not all features are available, however. Whenever possible, a CompatibilityError or warning will be raised when these features are encountered. Finally, api stability is not guarenteed for any aspect of the api.

The alpha period is an opportunity to test and experiment. Community engagement and feedback is highly valued, and will have an impact on future development. In the immediate future, work will focus on testing, stability, and basic configuration/validation. However, any feature ideas and feedback on the api are welcome. (Note that there's a number of issues I'm already aware of, so be sure to read this document before leaving bug reports).

Notable differences from pybids

Along with the substantial speed boost, rsbids optimizes many aspects of the pybids api:

Chained querying

rsbids.BidsLayout.get() returns a new instance of rsbids.BidsLayout. Calls to .get can thus be chained:

view = layout.get(suffix="T1w")

# later

view.get(subject="01")

Because of this, most of the methods in pybids.BIDSLayout can be replaced by an appropriate combination of methods:

# pybids
layout.get_subjects(suffix="events", task="stroop")

# rsbids
layout.get(suffix="events", task="stroop").entities["subject"]

# pybids
layout.get_files(scope="fmriprep")

# rsbids
layout.filter(scope="fmriprep")

# pybids
for f in layout.files:
    ...

# rsbids
for f in layout:
    ...

Simplified single-file querying

rsbids.BidsLayout has the .one property, which errors out if the layout does not have exactly one path. If more than one path is present, the entities still to be filtered are listed in the error:

# pybids (no error if more than one path)
layout.get(subject="001", session="02", suffix="dwi", extension=".nii.gz")[0]

# rsbids
layout.get(subject="001", session="02", suffix="dwi", extension=".nii.gz").one

Seperate .get() and .filter() methods

pybids uses the .get() method as an omnibus query method. While convenient, it makes the method brittle because certain arguments are interpreted with special meaning (e.g. scope, target). This makes it challenging to add additional query methods (e.g. searching specificially by pipeline or file root).

With the split, arguments to .get() will always be interpreted as entity names (e.g. subject, session, run, etc) or metadata keys (e.g. EchoTime, etc). All other special search modes are handled by .filter(). Because each query returns a new layout, it's perfectly possible to chain these calls together, making an extremely flexible query interface.

.get() accepts the "short" names of entities in addition to their long version. For instance, the following calls are equivalent:

layout.get(subject="001") == layout.get(sub=="001")

.filter() currently takes the following arguments:

root

Root searches by dataset root, making it useful for multi-root layouts. It accepts either the complete root as a string, or glob patterns (e.g. **/fmriprep-*).

scope

Scope uses the same syntax as in pybids: raw and self both match the raw dataset, derivatives matches all derivative datasets, <pipeline_name> searches derivative datasets by pipeline names found in their dataset_description.json.

Note that the above uses of scope are primarily included for backward compatibility with pybids. There are (or will be) better, dedicated ways to achieve each of these searches. Moving forward, scope will be intended to index labelled derivatives (see below).

Multi-root layouts

pybids supported single raw or root datasets with multiple, potentially nested derivative datasets. rsbids reimagines layouts as a flat collection of datasets, each tagged with various attributes. For example, one or more datasets may be raw, and the rest derivative. Datasets may be generated with one or more pipelines and derive from one or more datasets. These attributes are (or will be) individually indexed and individually queryable.

Thus, rsbids allows multiple raw roots:

# rsbids
layout = rsbids.layout(["root1", "root2"])

These roots can be then queried using roots:

layout.filter(root="root1")

New to rsbids, derivatives can be labelled:

#rsbids
layout = rsbids.layout(
    "dataset",
    derivatives={
        "proc1": "dataset/derivatives/proc1-v0.10.1",
        "anat": "dataset/derivatives/smriprep-v1.3",
    })

These labels can queried using scope:

layout.filter(scope="anat")

All derivatives can be selected using .derivatives:

layout.derivatives == layout.filter(scope="derivatives")

All dataset roots can be listed using with:

layout.roots

If the dataset has a single raw root (with any number of derivatives), the .root attribute can be used to retrieve that root:

layout = rsbids.layout(
    "dataset",
    derivatives={
        "proc1": "dataset/derivatives/proc1-v0.10.1",
        "anat": "dataset/derivatives/smriprep-v1.3",
    })

layout.root == "dataset"

If there is no raw root, but exactly one derivative root, .root will retrieve the derivative

layout = rsbids.layout(
    "dataset",
    derivatives={
        "proc1": "dataset/derivatives/proc1-v0.10.1",
        "anat": "dataset/derivatives/smriprep-v1.3",
    })
layout.filter(scope="proc1").root == "dataset/derivatives/proc1-v0.10.1"

All other calls to .root will error:

layout = rsbids.layout(
    "dataset",
    derivatives={
        "proc1": "dataset/derivatives/proc1-v0.10.1",
        "anat": "dataset/derivatives/smriprep-v1.3",
    })
layout.derivatives.root # !!! Error: multiple roots

The .description attribute works according to equivalent logic:

layout = rsbids.layout(
    "dataset",
    derivatives={
        "proc1": "dataset/derivatives/proc1-v0.10.1",
        "anat": "dataset/derivatives/smriprep-v1.3",
    })
layout.description == <DatasetDescription>

Note: The error handling for .description and .root is still a bit janky. DatasetDescription reading has only preliminary support: the object is readonly, and values must be accessed as attributes using snakecase:

layout.description.generated_by[0].name

layout.description["Name"] # !!! Error

Flexible parsing algorithm

rsbids has two variants of its parsing algorithm. One looks for entity-value pairs specifically defined by the bids spec (similar to how pybids and all other bids indexers currently work). Invalid entities (..._foobar-val_...) are ignored. This mode is enabled by rsbids.layout(..., validate=True), and gives a validation experience somewhat similar to pybids.BIDSLayout(..., validate=False, is_derivative=True) (note that this will change in the future to match the pybids defaults).

The other parser is completely generic: it parses any path looking for entity-value combinations seperated by underscores (_). So long as the path structure looks roughly bids-like, rsbids should correctly parse it, including missing extensions/suffixes, custom entities, any arbitrary value (so long as it has no _), custom datatypes, malformed directory structures, etc.

The flexible algorithm currently has no validation, so any path will be parsed into something according to the algorithm. In the future, rsbids will allow for more fine-grained validation.

The details of the algorithm will be written at some point in the future. In summary, these are the main priorities:

1. Any valid bids path MUST be parsed correctly (if it's not, it's a bug)
2. Any almost-valid bids path SHOULD be parsed correctly. This include paths with one or a few of:

• Custom entity
• Custom datatype
• Custom entity as a directory if it's also in the file name

Finally, any path bits that can't be interpreted as key-value pairs will generally be saved as parts (e.g. sub-001_somepart_ses-1_...). In the future, rsbids will supporting querying for these parts, making it potentially useful even for severely non-bids-compliant datasets.