marspylib 0.3.0

Mars - Molecule ARchive Suite

Pure-Python library for reading and writing Mars Molecule Archives (.yama) and utility functions for working with them — no JVM, no Fiji install required. Complete Molecule ARchive Suite (Mars) documentation including a guide to working with mars data structures in python can be found at mars-docs.

Installation

pip install marspylib

This project should soon be available for installation through conda forge.

Only dependency requirements are numpy, pandas, and matplotlib — this package can be installed in any plain Python/conda environment and does not require Fiji, ImageJ, or a JVM of any kind.

Reading/writing archives on S3 (see Using S3 below) needs boto3, kept as an optional extra so it's not a hard dependency for users who only ever work with local files:

pip install marspylib[s3]

Usage

import marspylib.yama as yama

archive = yama.open("experiment.yama")
print(archive.properties.number_of_molecules)

for molecule in archive:
    df = molecule.table            # pandas.DataFrame
    if "accepted" in molecule.tags:
        ...

molecule = archive["some-uid"]     # random access by UID

archive.save("experiment_out.yama")

Creating a new archive

You don't need Fiji to build a .yama from scratch — construct Properties, a Molecule (or one of its subclasses, matching the archive type — see Supported archive types), and MarsMetadata/regions/positions as needed, then put() each record into a fresh Archive and save:

import pandas as pd
import marspylib.yama as yama

properties = yama.Properties(archive_type=yama.ARCHIVE_TYPES["SingleMoleculeArchive"])
archive = yama.Archive(properties, metadata={}, molecules={})

metadata = yama.MarsMetadata(microscope="Nikon Ti2", source_directory="/data/2024-01-15")
archive.put_metadata(metadata)

for i in range(3):
    molecule = yama.SingleMolecule(metadata_uid=metadata.uid)
    molecule.add_tag("accepted")
    molecule.parameters["dwell"] = 5.5
    molecule.table = pd.DataFrame({
        "T": [0.0, 1.0, 2.0],
        "Intensity": [10.1 + i, 10.5 + i, 10.9 + i],
    })
    archive.put(molecule)   # molecule.uid was auto-generated -- see below

archive.save("new_experiment.yama")

Leaving uid unset when constructing a Molecule/MarsMetadata (as above) auto-generates one in mars-core's own format — the same Base58 encoding of a random UUID that Fiji itself uses (MarsMath.getUUID58()), so records you create in Python get UIDs that are unique right alongside ones created in Fiji, which matters if the two ever get merged into the same archive. You can also call the generator directly, or supply your own uid=:

yama.new_molecule_uid()    # e.g. "8eoHfZ1GdvNBeWGNNDY3hp" -- full-length
yama.new_metadata_uid()    # e.g. "mnAgQYYn63" -- fixed 10 characters, matching mars-core

molecule = yama.SingleMolecule(uid="my-own-id")   # or supply your own

yama.ARCHIVE_TYPES maps every supported short name (SingleMoleculeArchive, DnaMoleculeArchive, DefaultMoleculeArchive, ObjectArchive, TransverseFlowArchive) to the archive-type string mars-core expects — use whichever matches the Molecule subclass you're building records with.

Removing records

archive.remove("some-uid")           # drop a molecule
archive.remove_metadata("meta-uid")  # drop a metadata record
archive.save()

For a .yama.store archive, remove()/remove_metadata() delete the underlying file immediately (matching mars-core), not deferred to the next save(); for a single-file archive it just drops the record from memory, taking effect the next time you save().

Virtual archives (.yama.store)

Large archives saved from Fiji as a .yama.store directory (rather than a single .yama file) open the same way — yama.open() detects it from the path — but records are loaded lazily, one at a time, instead of all at once:

archive = yama.open("experiment.yama.store")   # nothing loaded yet
print(len(archive), "molecules")

for molecule in archive:                        # each one read from disk as you reach it
    if "accepted" in molecule.tags:
        ...

molecule = archive["some-uid"]                  # read once, then cached for reuse

Writing works the same archive.save(...) call as single-file archives, dispatched purely on whether the target path ends in .yama.store:

archive["some-uid"].add_tag("reviewed")
archive.save()                          # update the .yama.store in place

archive.save("subset.yama")             # flatten into a single file instead
archive.save("copy.yama.store")         # write out as a separate virtual store

A .yama.store archive must still exist on disk for as long as you're reading from it — a molecule you haven't touched yet is only read from its file the moment you access it, so don't delete, move, or let a temporary directory holding one go out of scope while you're still using the archive.

Molecules in a .yama.store are loaded lazily and LRU-cached (default 128 at a time) — mutating one you already hold a reference to (like molecule.add_tag(...) above) is safe as long as it's still cached when you call save(). If you're touching more distinct molecules than that between editing one and saving, or want to be certain, use archive.put(molecule) to pin it so it's guaranteed to be written regardless of cache pressure:

molecule = archive["some-uid"]
molecule.add_tag("reviewed")
archive.put(molecule)     # pins it -- guaranteed to persist even if evicted
# ... touch hundreds of other molecules ...
archive.save()

put()/put_metadata() are also how you add a brand-new record (a UID that wasn't already in the archive) — see the mapping table below.

Using S3

Archives (single .yama files or .yama.store virtual archives) can be opened and saved directly on S3 or any S3-compatible endpoint, with no local copy needed — open_s3()/archive.save_s3() mirror open()/archive.save() exactly, including the lazy loading and immediate-delete-on-remove() behavior for .yama.store.

A location is three things: the endpoint host (server_address), the bucket, and the key (path to the file or .yama.store directory within the bucket). Passing these three separately is the recommended way — it's unambiguous, unlike a combined URL, which can be misread if a bucket or host name happens to contain a stray .:

import marspylib.yama as yama

archive = yama.open_s3(server_address="storage.example.org",
                        bucket="my-bucket", key="path/to/experiment.yama")

archive["some-uid"].add_tag("reviewed")
archive.save()   # no args -- saves back to the same S3 location it was opened from

archive.save_s3(server_address="storage.example.org",
                 bucket="my-bucket", key="path/to/experiment.yama.store")

server_address defaults to https; for a plain-http (no TLS) endpoint, either pass secure=False or just include the scheme directly in server_address ("https://..."/"http://..." are both accepted and override secure accordingly) — whichever you find easier to read.

If you already work with combined virtual-hosted-style URLs (https://<bucket>.s3.<server_address>/<key>), those are also supported, as a single location argument in place of the three separate fields:

archive = yama.open_s3("https://my-bucket.s3.storage.example.org/path/to/experiment.yama")
archive.save_s3("https://my-bucket.s3.storage.example.org/path/to/experiment.yama.store")

The .s3. in that combined form is a fixed separator token, not necessarily part of the real endpoint host — it's stripped back off when parsed, so both forms above point at the same server_address="storage.example.org". (Real AWS S3 is a case where the actual endpoint host, s3.amazonaws.com, separately happens to also start with s3. — that's a coincidence of AWS's own naming, not something this parsing depends on.)

Both forms work with .yama.store too, dispatched the same way save() dispatches locally — by whether key ends in .yama.store.

Credentials are never passed through this API directly. By default, open_s3()/save_s3() resolve credentials the standard boto3 way — environment variables, ~/.aws/credentials, SSO cache, IAM role — the same chain the AWS SDK for Java (and so Fiji/mars-core) uses, so if your credentials are already configured locally, nothing further needs to be set up here. To use a specific profile or override resolution, pass your own boto3.Session:

import boto3

session = boto3.Session(profile_name="my-profile")
archive = yama.open_s3(server_address="storage.example.org", bucket="my-bucket",
                        key="path/to/experiment.yama", session=session)

Supported archive types

Every SingleMoleculeArchive, DnaMoleculeArchive, and DefaultMoleculeArchive opened from Fiji gets its own Molecule subclass (SingleMolecule, DnaMolecule, DefaultMolecule) — currently identical to plain Molecule, since in mars-core these three only differ in name, not in the fields they store today. Every archive type gets a dedicated Python class uniformly, even when (as here) there's nothing archive-type-specific about it yet, so a future mars-core field addition to any one of them only means filling in that one class here, not restructuring anything.

Two other archive types carry one extra field per record, so they get their own Molecule subclasses:

ObjectArchive (mars-core's object package) uses MartianObject, which adds .shapes: a dict[int, PeakShape] mapping timepoint → tracked polygon outline.

archive = yama.open("tracked_objects.yama")   # archive_type is ObjectArchive
obj = archive["some-uid"]
shape = obj.shapes[12]        # PeakShape at timepoint 12
shape.x, shape.y              # coordinate arrays (same length)
obj.shapes[13] = yama.PeakShape(x=[...], y=[...])
archive.save()

TransverseFlowArchive (the separate mars-transverseflow module) uses TransverseFlowMolecule, which adds .replication_fork_shapes: a dict[int, ReplicationForkShape] mapping timepoint → replication fork geometry (parental/leading/lagging strand outlines, each with an optional per-channel intensity profile).

archive = yama.open("forks.yama")   # archive_type is TransverseFlowArchive
mol = archive["some-uid"]
shape = mol.replication_fork_shapes[7]
shape.parental_x, shape.parental_y     # not-yet-replicated duplex outline
shape.leading_x, shape.leading_y       # leading-strand daughter outline
shape.lagging_x, shape.lagging_y       # lagging-strand daughter outline
shape.leading_intensity["GFP"][7]      # per-channel intensity, keyed by coordinate

Coming from Java/Groovy/Fiji: method name mapping

This library's classes cover the same data as mars-core's Java classes, but follow Python naming conventions (snake_case, attributes instead of getters/setters) rather than mirroring the Java API name-for-name. This applies equally to Groovy scripts written against Mars in Fiji's script editor — Groovy calls Java methods with identical syntax (molecule.addTag(tag) is valid in both), so everywhere this table says "Java" it means Groovy too. Where a Java/Groovy call was a getter/setter pair, the Python side is usually just a plain attribute you can read or assign directly.

Archive (de.mpg.biochem.mars.molecule.MoleculeArchive)

Java	Python
archive.get(uid)	archive[uid]
archive.getMetadata(uid)	archive.get_metadata(uid)
archive.metadataHasTag(uid, tag)	archive.metadata_has_tag(uid, tag)
archive.getNumberOfMolecules()	len(archive) or archive.properties.number_of_molecules
archive.getNumberOfMetadatas()	len(list(archive.metadata))
archive.molecules().forEach(...)	for molecule in archive: ...
archive.getMolecules() iteration	for molecule in archive: ...
(metadata collection)	archive.metadata (iterator)
archive.properties()	archive.properties (attribute)
archive.save()	archive.save()
archive.saveAs(file)	archive.save(path)
archive.saveAsVirtualStore(dir)	archive.save(path) where path ends in .yama.store
archive.put(molecule)	archive.put(molecule)
archive.putMetadata(m)	archive.put_metadata(m)
archive.remove(uid) / remove(molecule)	archive.remove(uid)
archive.removeMetadata(uid) / removeMetadata(m)	archive.remove_metadata(uid)
archive.contains(uid)	uid in archive
archive.containsMetadata(uid)	archive.contains_metadata(uid)
archive.getMoleculeUIDs()	archive.molecule_uids()
archive.getMetadataUIDs()	archive.metadata_uids()
archive.isVirtual()	archive.is_virtual (attribute)
archive.getComments() / setComments(s)	archive.get_comments() / archive.set_comments(s)
archive.getTagList(uid) / getTagSet(uid)	archive.molecule_tags(uid)
archive.getChannel(uid)	archive.molecule_channel(uid)
archive.getImage(uid)	archive.molecule_image(uid)
archive.getMetadataUIDforMolecule(uid)	archive.molecule_metadata_uid(uid)
archive.moleculeHasTag(uid, tag)	archive.molecule_has_tag(uid, tag)
archive.moleculeHasTags(uid) / moleculeHasNoTags(uid)	archive.molecule_has_tags(uid)
archive.getMetadataTagList(uid) / getMetadataTagSet(uid)	archive.metadata_tags(uid)

The molecule_*/metadata_* lookups above are the same "don't load the full record just to check one field" trick Java gets from its index: for a .yama.store opened with an indexes.sml present, they answer from that index alone (tags/channel/image/metadata linkage only — not parameters, regions, positions, or the table, which aren't indexed in mars-core either). For a single-file archive, or a record you've already loaded/put() in this session, they just read the in-memory object directly, which is equally cheap. archive[uid].tags etc. still works too, of course — it's just not free for a not-yet-touched .yama.store record the way these are.

Molecule / MarsRecord (de.mpg.biochem.mars.molecule.Molecule, MarsRecord)

Java	Python
molecule.getUID()	molecule.uid
molecule.getNotes() / setNotes(s)	molecule.notes
molecule.getTags()	molecule.tags (list)
molecule.addTag(tag)	molecule.add_tag(tag)
molecule.hasTag(tag)	molecule.has_tag(tag)
molecule.getParameter(name)	molecule.parameters[name]
molecule.setParameter(name, v)	molecule.parameters[name] = v
molecule.hasParameter(name)	name in molecule.parameters
molecule.getRegion(name) / putRegion(r)	molecule.regions[name]
molecule.hasRegion(name)	molecule.has_region(name)
molecule.getPosition(name) / putPosition(p)	molecule.positions[name]
molecule.hasPosition(name)	molecule.has_position(name)
molecule.getTable()	molecule.table (pandas.DataFrame)
molecule.getMetadataUID() / setMetadataUID(s)	molecule.metadata_uid
molecule.getImage() / setImage(i)	molecule.image
molecule.getChannel() / setChannel(c)	molecule.channel
molecule.getSegmentsTable(x, y, region)	molecule.segment_tables[(x, y, region)]

MarsMetadata (de.mpg.biochem.mars.metadata.MarsMetadata)

Java	Python
metadata.getUID()	metadata.uid
metadata.getMicroscopeName() / setMicroscopeName(s)	metadata.microscope
metadata.getSourceDirectory() / setSourceDirectory(s)	metadata.source_directory
metadata.getLog()	metadata.log
metadata.hasTag(tag)	metadata.has_tag(tag)
metadata.getBdvSource(name)	metadata.bdv_sources[name]

MarsRegion / MarsPosition (de.mpg.biochem.mars.util)

Java	Python
region.getName() / getColumn() / getStart() / getEnd() / getColor() / getOpacity()	.name / .column / .start / .end / .color / .opacity
position.getName() / getColumn() / getPosition() / getColor() / getStroke()	.name / .column / .position / .color / .stroke

MoleculeArchiveProperties

Java	Python
properties.getSchema() / getInputSchema()	properties.schema
properties.getTagSet()	properties.tag_set
properties.getChannelSet()	properties.channel_set
properties.getColumnSet()	properties.table_column_set
properties.getParameterSet()	properties.parameter_set
properties.getRegionSet()	properties.region_set
properties.getPositionSet()	properties.position_set
properties.getSegmentsTableNames()	properties.segment_table_names
document access	properties.documents[name]

Note archive.save() recomputes properties.tag_set/channel_set/ parameter_set/region_set/position_set/table_column_set/ segment_table_names/number_of_molecules/number_of_metadata from the archive's actual current contents before writing (mirroring mars-core's own rebuildIndexes()), so these always reflect reality after a put() even though nothing updates them incrementally as you go.