homr 0.2.0

End-to-end Optical Music Recognition (OMR) system build on top of vision transformers.

homr

homr is an Optical Music Recognition (OMR) software designed to transform camera pictures of sheet music into machine-readable MusicXML format. The resulting MusicXML files can be further processed using tools such as musescore.

Prequisites

• Python 3.10
• Poetry
• Optional: NVidia GPU with CUDA 12.1

Getting started

• Clone the repository
• Install dependencies using poetry install
• Run the program using poetry run homr <image>
• The resulting MusicXML file will be saved in the same directory as the input image

Example

The example below provides an overview of the current performance of the implementation. While some errors are present in the output, the overall structure remains accurate.

Original Image	homr Result

The homr result is obtained by processing the homr output and rendering it with musescore.

Technical Details

homr employs segmentation techniques outlined in oemer to identify staff lines, clefs, bar lines, and note heads in an image. These components are combined to determine the position of staffs within the picture.

Subsequently, each staff image undergoes transformation using a transformer model (based on Polyphonic-TrOMR) to identify symbols present on the staff. Pitch information is cross-validated with note head data obtained from the segmentation model.

The results are then converted into MusicXML format and saved to disk.

Image Predictions

homr utilizes oemer's UNet implementations to isolate staff lines and other symbols for note head identification. These predictions serve as input for staff and symbol detection.

Preprocessing the image has shown to enhance robustness against noisy backgrounds and variations in brightness.

Staff and Symbol Detection

The detection process involves extracting model data types from the image predictions. A key concept is the "staff anchor," which serves as a reference point ensuring accurate staff detection amidst symbols that might obscure it. Clefs and bar lines are currently utilized as anchor symbols.

For each anchor, the algorithm attempts to locate five staff lines and constructs the remainder of the staff around these anchors.

Unit Sizes

The unit size denotes the distance between staff lines, which may vary due to camera perspective. To accommodate this, the unit size is calculated per staff.

Connecting Staffs

Support for multiple voices and grand staffs is facilitated by identifying braces and brackets to combine individual staffs.

Rhythm Parsing

Dewarped images of each staff are computed and passed through a transformer to extract staff contents. From this point onward, semantic information from the sheet music is utilized rather than pixel-based data.

XML Generation

The previous outputs in terms of result model objects are used to generate music XML.

Citation

If you use this code in your research work, please cite oemer and Polyphonic-TrOMR.

Name

The name "homr" stands for Homer's Optical Music Recognition (OMR), leaving the interpretation of "Homer" to the user's discretion, whether referring to the ancient poet Homer or the iconic character from The Simpsons.

Thanks

This project builds upon previous work, including:

• The segmentation models of oemer
• The transformer model of Polyphonic-TrOMR
• The starter template provided by Benjamin Roland