arfx 3.0.0

input files

arfx can read sampled data from pcm, wave, npy and mda files. Support for additional file formats can be added as plugins (see 4).

When adding data to an ARF container (-c and -r modes), the input files are specified on the command line, and added in the order given. By default, entries are given the same name as the input file, minus the extension; however, if the input file has more than one entry, they are given an additional numerical extension. To override this, the -n flag can be used to specify the base name; all entries are given sequential names based on this.

The -n, -a, -e, -p, -s, -T options are used to store information about the data being added to the file. The DATATYPE argument can be the numerical code or enumeration code (run arfx --help-datatypes for a list), and indicates the type of data in the entries. All of the entries created in a single run of arfx are given these values. The -u option tells arfx not to compress the data, which can speed up I/O operations slightly.

Currently only one sampled dataset per entry is supported. Clearly this does not encompass many use cases, but arfx is intended as a simple tool. More specialized import procedures can be easily written in Python using the arf library.

output files

The entries to be extracted (in -x mode) can be specified by name. If no names are specified, all the entries are extracted. All sampled datasets in each entry are extracted as separate channels, because they may have different sampling rates. Event datasets are not extracted.

By default the output files will be in wave format and will have names with the format entry_channel.wav. The -n argument can be used to customize the names and file format of the output files. The argument must be a template in the format defined by the python string module. Supported field names include entry, channel, and index, as well as the names of any HDF5 attributes stored on the entry or channel. The extension of the output template is used to determine the file format. Currently only wave is supported, but additional formats may be supplied as plugins (see below).

The metadata options are ignored when extracting files; any metadata present in the ARF container that is also supported by the target container is copied.

other operations

As with tar, the -t operation will list the contents of the archive. Each entry/channel is listed on a separate line in path notation.

The -A flag is used to copy the contents of one ARF file to another. The entries are copied without modification from the source ARF file(s) to the target container.

The -d (delete) operation uses the same syntax as the extract operation, but instead of extracting the entries, they are deleted. Because of limitations in the underlying HDF5 library, this does not free up the space; pass the -P option to repack the file afterwards and reclaim it. Repacking requires h5repack on the path.

The -U (update) operation can be used to add or update attributes of entries.

The --write-attr operation can be used to store the contents of text files in top-level attributes. The attributes have the name user_<filename>. The --read-attr operation can be used to read out those attributes. This is useful when data collection programs generate log or settings files that you want to store in the ARF file.

arfx-split

This script is used to reorganize very large recordings, possibly contained in multiple files, into manageable chunks. Each new entry is given an updated timestamp and attributes from the source entries. Only sampled datasets are processed.

Entries that are contiguous on the sample timeline will be spliced back together before chunking as long as there some internal track of the sample count, such as the jack_frame attribute recorded by jrecord. This is an unsigned 32-bit value that wraps roughly every 27 hours at typical audio sampling rates, triggering the start of a new entry in the ARF file.

• –no-splice: chunk each source entry separately, as before

• –frame-attr NAME: use a different attribute as the frame counter

• –max-overlap SAMPLES: splice entries that overlap by up to this many samples, dropping the duplicates (default 4096). The overlapping samples are compared first, and entries whose data disagree there are left unspliced and reported, since the counter and the data cannot both be right.

arfx-oephys

Converts the output of an open-ephys recording (open-ephys binary format) into an ARF file. open-ephys stores its data in a big complex directory tree, which this script will navigate and store in an appropriately timestamped entry in the ARF file. Has not been tested with data from outside our lab. Example invocation:

arfx-oephys -T EXTRAC_HP -k experimenter=smm3rc -k bird=C194 -k pen=1 -k site=1 -k protocol=chorus -f C194_1_1.arf C194_2023-10-16_16-30-54_chorus/

We typically run this command before starting spike sorting to create a copy of the recording for archival.

From GUI 0.6 on, a session directory may hold more than one experimentN/recordingM. Each becomes its own entry, named for the session and its place within it, and each is timestamped from the wall-clock time in its own sync_messages.txt rather than from the session directory name, which gives only the start of the first recording. Older recordings have one recording per directory and no wall-clock time recorded, so they take the timestamp from the directory name.

arfx-collect-sampled

This script is used to export data into a flat binary structure. It collects sampled data across channels and entries into a single 2-D array. The output can be stored in a multichannel wav file or in a raw binary dat format (N samples by M channels), which is used by a wide variety of spike-sorting tools. We use this script if we ever have to re-sort a recording after deleting the original raw recording.

arfx-select

This is a pretty specialized script that takes in a table of segments defined by entry name and start/stop time and copies them to a new ARF file. It’s usually better to just write analysis code to directly access the desired data from the original file, but it can be useful as a first stage in exporting small segments of a recording to wave files for sharing or depositing.