nnfasta 0.1.18

Neural Net efficient FASTA

nnfasta

Lightweight Neural Net efficient FASTA dataset suitable for training.

Should be memory efficient across process boundaries. So useful as input to torch/tensorflow dataloaders with multiple workers etc.

Presents a list of fasta files as a simple abc.Sequence so you can inquire about len(dataset) and retrieve Records randomly with dataset[i]

Uses Python's mmap.mmap under the hood.

The underlying FASTA's should be "well formed" since there is minimal sanity checking done.

Install

Install with:

pip install nnfasta

There are no dependencies, you just need a modern (>= 3.9) python.

Usage

from nnfasta import nnfastas

dataset = nnfastas(['athaliana.fasta','triticum.fasta','zmays.fasta'])

# display the number of sequences
print(len(dataset))

# get a particular record
rec = dataset[20]
print('sequence', rec.id, rec.description, rec.seq)

Warning: No checks are made for the existence of the fasta files. Also files of zero length will be rejected by mmap.

A Record mimics biopython's SeqRecord and is simply:

@dataclass
class Record:
    id: str
    """Sequence ID"""
    description: str
    """Line prefixed by '>'"""
    seq: str
    """Sequence stripped of whitespace and uppercased"""

    @property
    def name(self) -> str:
        return self.id

Arguments

You can give nnfastas either a filename, a Path, the actual bytes in the file or an open file pointer (opened with mode="rb") OR a list of these things.

Encoding

The files are assumed to be encoded as "ASCII". If this is not the case the nnfastas accepts an encoding argument. All the files presented to nnfastas are assumed to be similarly encoded.

Test and Train Split best practice

Use SubsetFasta

from nnfasta import nnfastas, SubsetFasta
from sklearn.model_selection import train_test_split

dataset = nnfastas(['athaliana.fasta','triticum.fasta','zmays.fasta'])
train_idx, test_idx = train_test_split(range(len(dataset)),test_size=.1,shuffle=True)

# these are still Sequence[Record] objects.

train_data = SubsetFasta(dataset, train_idx)
test_data = SubsetFasta(dataset, test_idx)

# *OR* ... this is basically the same
import torch
train_data, test_data = torch.utils.data.random_split(dataset, [.9, .1])

See the pytorch Subset logic here

Enjoy peps!