shira-audio 0.1.1

audio search/retrieval library

shira 🔖🎧

A simple audio search/retrieval library. (wip)

This is the audio version of ripple. Search through audio files/data with text queries or audio samples.
It's meant to be an neural encoded version of Shazam, but might just be for small scale/local usage.

Methodology

It's basically a semantic search library for audio.

The local audio data/files are indexed and embeddings are generated(with CLAP), then a FAISS vector index is created.
The files are retrieved based on cosine similarity between embeddings.

This process makes use of contrastively pretrained audio-language model, CLAP(like OpenAI CLIP for audio), specifically LAION's laion/larger_clap_music_and_speech checkpoint/model

usage

• Install the library

pip install shira-audio

• For text-based search

from shira import AudioSearch, AudioEmbedding

embedder = AudioEmbedding(data_path='.') # init embedder class
audio_data_embeds = embedder.index_files() # create embeddings and index audio files

neural_search = AudioSearch() # init semantic search class

text_query = 'classical music' # text description for search

# get k similar audio w/probability score pairs 
matching_samples, scores = neural_search.text_search(text_query, audio_data_embeds, k_count=5)

matching_samples[0]['audio']['path'] # get file path for the top sample

Acknowldgements

• CLAP: Learning Audio Concepts From Natural Language Supervision
• tinyCLAP: Distilling Contrastive Language-Audio Pretrained models
• Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation
• laion/larger_clap_music_and_speech model by LAION.