Module for finetuning dfm base-models to sentence transformers
Project description
dfm-sentence-transformers
Code for curating data and training sentence transformers for the Danish Foundation Models project.
Training
Install the CLI:
pip install dfm_sentence_trf
WARNING: The package is not on PyPI yet, so this won't actually work as of yet.
Config system (TODO)
You have to specify basic model and training parameters, as well as all the tasks/datasets the model should be trained on.
[model]
name="dfm-sentence-encoder-small-v1"
base_model="chcaa/dfm-encoder-small-v1"
device="cpu"
[training]
epochs=5
warmup_steps=100
batch_size=120
[tasks]
[tasks.bornholmsk]
@tasks="multiple_negatives_ranking"
sentence1="da_bornholm"
sentence2="da"
[tasks.bornholmsk.dataset]
@loaders="load_dataset"
path="strombergnlp/bornholmsk_parallel"
[tasks.hestenet]
@tasks="multiple_negatives_ranking"
sentence1="question"
sentence2="answer"
[tasks.hestenet.dataset]
@loaders="load_dataset"
path="some/local/path"
Then you can train a sentence transformer by using the finetune
command.
python3 -m dfm_sentence_trf training.cfg --output_folder "output/"
You can push the finetuned model to HuggingFace Hub:
python3 -m dfm_sentence_trf training.cfg --model_path "output/" --organization "chcaa"
Tasks (TODO)
ContrastiveParallel (TODO)
The task uses a contrastive loss on a parallel corpus, where negative examples (aka. non-matching sentence pairs labelled with 0) are randomly sampled. You can specify the dataset, and the number of negative samples for each positive sample. As well as basic training parameters.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for dfm_sentence_transformers-0.1.0.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | ca6649928f96b85c6a3c9634717c3226980dc51f6a7c6accb62401d50aee2453 |
|
MD5 | 642764c12ab5cd0e2fc7abc7e2e224bf |
|
BLAKE2b-256 | 7c97f5bf7b6482c11c8ab9a68d6c5e763d366a4096dfb127a9c869e12d3fad5f |
Hashes for dfm_sentence_transformers-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 8adbbfb3958273592568ce0e05ab25f92ec8dc94b8e84b41df72781975ebee8c |
|
MD5 | d38c3e0fc8a6ba601a9e368b16e1ddff |
|
BLAKE2b-256 | 6ee87c1c665f0c1cfad4f9315689992ee1d6c39a48807b81fcc036883a8d4eff |