Choosing an Embedding Loss Function • Rutvik Acharya

In Fine-Tuning Embeddings with Contrastive Learning and MRL I covered why you would fine-tune an embedding model. This post is about the decision that comes right after: which loss function.

sentence-transformers ships close to thirty of them, and the docs list them without saying much about how they behave differently. The short version is that the loss is where you define what “similar” means. Two losses trained on the same texts can produce models that rank, cluster and score very differently.

The best way to feel that is to train them. Every loss below has a ▶ Train button that loads it into a playground further down the page. The playground fine-tunes a small embedding model live in your browser and shows what the loss is doing inside each batch.

Your data picks the loss#

Before comparing formulas, look at one row of your training data. Its shape rules most losses in or out.

(anchor, positive) Most common

anchor: i want my money back
positive: Refund request process for returned orders

Start with MultipleNegativesRankingLoss

CachedMNRL for big batches · GISTEmbedLoss to filter false negatives

(anchor, positive, negative)

anchor: i want my money back
positive: Refund request process…
negative: Cancel subscription plan

Start with MNRL with hard negatives

TripletLoss when you want an explicit margin

(sentence1, sentence2, score)

sentence1: battery goes for days
sentence2: Battery drains too quickly
score: 0.4

Start with CoSENTLoss

AnglELoss · CosineSimilarityLoss as a simple baseline

(text, label)

text: vpn wont connect from home
label: 0 (Network)

Start with BatchHardTripletLoss

BatchAllTriplet · BatchSemiHardTriplet · BatchHardSoftMargin

If you only have questions and the passages that answer them, which is the usual case for RAG, you are in the top-left card. Most teams are.

What every loss is doing#

Every loss in this post computes cosine similarities between embeddings and then pushes some of them up and others down. They differ in three ways:

Family	What gets compared	When a row stops teaching
Ranking (MNRL)	Each anchor against every positive in the batch	Almost never: the softmax always wants more separation
Triplet	Anchor vs. its own positive and its own negative	As soon as the margin is met
Scored similarity	Each pair’s cosine against a gold score	CosineSim: when cosine equals the score. CoSENT: when the order is right
Label-based triplet	Every text against every other text in the batch, by label	When all same-label texts beat all other-label texts by the margin

Keep that last column in mind. It explains most of the behaviour you will see in the playground.

Ranking: MultipleNegativesRankingLoss#

MNRL (also called InfoNCE or in-batch softmax loss) is the default for retrieval. You give it (anchor, positive) pairs. For a batch of $B$ pairs, it treats each anchor as a $B$ -way classification problem: which of the $B$ positives in this batch is mine?

\mathcal{L} = -\frac{1}{B}\sum_{i=1}^{B} \log \frac{\exp\big(s \cdot \cos(a_i, p_i)\big)}{\sum_{j=1}^{B} \exp\big(s \cdot \cos(a_i, p_j)\big)}

In plain words

Think of it as a multiple-choice quiz. For each question in the batch, the model has to pick its correct passage out of all the passages in the batch. The loss is small when the model puts nearly all its confidence on the right one.

$a_i$ is question $i$ , $p_i$ is its correct passage, and $p_j$ runs over every passage in the batch.
$\cos$ is the cosine similarity: how closely two embeddings point in the same direction.
The fraction turns those similarities into probabilities (a softmax) that add up to 1. The top is the right answer; the bottom is all the options.
$-\log$ of a probability is close to 0 when the probability is near 1, and grows fast as it drops.
$s$ (the scale) stretches the differences so that small similarity gaps become big probability gaps.

Worked example. “i want my money back” is compared with three passages, with $s = 20$ :

Passage	cos	× 20	Probability
Refund process (correct)	0.60	12	0.881
Track shipment	0.50	10	0.119
Reset password	0.10	2	0.00004

Loss $= -\log(0.881) = 0.127$ . Most of the leftover loss comes from “Track shipment” sitting too close, so training pushes that one away. Once the correct passage reaches cosine 0.8 and “Track shipment” falls to 0.3, the loss drops to about 0.00005.

The other rows’ positives act as negatives for free. That has three practical consequences:

Batch size is a hyperparameter of the loss. A batch of 64 gives every anchor 63 negatives; a batch of 4 gives it 3. Use the largest batch that fits in memory. CachedMultipleNegativesRankingLoss uses gradient caching to reach batch sizes in the thousands on one GPU.
Duplicates poison the batch. If two rows in a batch share a positive, one anchor is told to push away its own correct answer. Use BatchSamplers.NO_DUPLICATES so that can’t happen.
Scale is an inverse temperature. The default scale=20 (temperature 0.05) makes the softmax sharp, so it focuses on the negatives that are closest to the anchor.

If you have mined hard negatives, add them as a third column. MNRL then scores each anchor against all in-batch positives plus every hard negative, which is usually the strongest setup for retrieval. GISTEmbedLoss goes one step further and uses a guide model to drop in-batch “negatives” that are actually relevant.

Triplet: TripletLoss#

TripletLoss takes (anchor, positive, negative) and asks for one thing: the anchor must be closer to the positive than to the negative by at least a margin $m$ .

\mathcal{L} = \max\big(0,\; d(a, p) - d(a, n) + m\big), \qquad d = 1 - \cos

In plain words

The correct passage has to be closer than the wrong one, and by a clear gap rather than a hair. If it already is, the loss is zero and nothing changes.

$d$ is a distance: $1 - \cos$ , so 0 means identical and bigger means further apart.
$d(a,p)$ is how far the question is from the right passage; $d(a,n)$ is how far it is from the wrong one.
$m$ is the margin, the minimum gap you demand.
$\max(0, \dots)$ means the loss can never go below zero: once the gap is big enough, the triplet is “done”.

Worked example with margin 0.3. The question has cosine 0.55 to the right passage and 0.40 to the wrong one, so the distances are 0.45 and 0.60:

$0.45 - 0.60 + 0.3 = 0.15$ , so the loss is 0.15 and training pulls the right passage closer.

After training the right passage is at cosine 0.80 (distance 0.20): $0.20 - 0.60 + 0.3 = -0.10$ , which is below zero, so the loss is 0 and this triplet stops teaching.

The $\max(0, \cdot)$ is the important part. Once a triplet satisfies the margin, its loss and gradient are exactly zero and it stops contributing. That makes TripletLoss stable and easy to reason about, but it also means it learns less from each row than MNRL, which keeps pushing.

Scored similarity: CosineSimilarityLoss vs. CoSENT#

When each pair comes with a graded score (1.0 same meaning, 0.4 related, 0.0 unrelated), you have two philosophies.

CosineSimilarityLoss is plain regression. It pushes each pair’s cosine onto its label:

\mathcal{L} = \frac{1}{B}\sum_{k} \big(\cos(u_k, v_k) - y_k\big)^2

In plain words

Each pair comes with a target score. The loss measures how far the model’s cosine is from that target, squares it, and averages over the batch.

$u_k, v_k$ are the two texts in pair $k$ , and $y_k$ is its gold score.
Squaring makes big misses cost much more than small ones.
$\frac{1}{B}\sum$ is just the average over the $B$ pairs in the batch.

Worked example. A pair labelled 0.4 that the model scores at 0.7 costs $(0.7 - 0.4)^2 = 0.09$ . A pair labelled 1.0 scored at 0.9 costs only $(0.9 - 1.0)^2 = 0.01$ . The first pair gets pushed down toward 0.4 even if it is already ranked correctly against the others.

CoSENTLoss only cares about order. For every two pairs in the batch where pair $k$ has a higher gold score than pair $l$ , it penalises the model if $l$ ‘s cosine is higher:

\mathcal{L} = \log\Big(1 + \sum_{y_k > y_l} \exp\big(s \cdot (\cos_l - \cos_k)\big)\Big)

In plain words

Take every two pairs where one has a higher gold score than the other. If the model ranks them the wrong way round, add a penalty that grows quickly with how wrong it is. The exact cosine values don’t matter, only the order.

The sum runs over every combination where pair $k$ should outrank pair $l$ ( $y_k > y_l$ ).
$\cos_l - \cos_k$ is positive when the lower-scored pair looks more similar, which is the mistake.
$\exp$ turns that into a penalty that is tiny when the order is right and large when it is wrong. $s$ makes the switch sharper.
$\log(1 + \dots)$ keeps the total well-behaved: close to 0 when everything is in order.

Worked example with $s = 20$ . Pair A is labelled 1.0 but has cosine 0.6; pair B is labelled 0.4 but has cosine 0.7, so they’re in the wrong order:

$\log\big(1 + e^{20 \times (0.7 - 0.6)}\big) = \log(1 + e^{2}) = 2.13$

Once training flips them (A at 0.8, B at 0.5): $\log\big(1 + e^{20 \times (0.5 - 0.8)}\big) = \log(1 + e^{-6}) = 0.0025$ . It never needed A to reach exactly 1.0 or B to reach exactly 0.4.

CoSENT usually wins on real STS benchmarks because human scores are noisy and live on their own scale (often 0 to 5). Cosine does not have to equal “0.4”, it only has to rank the pair correctly. AnglELoss is a variant of CoSENT that optimises angle differences instead, which avoids the flat gradient cosine has near ±1.

Labels only: batch triplet losses#

Sometimes you have no pairs at all, just texts with class labels: tickets and their queue, products and their category. The batch triplet losses build triplets for you inside each batch: any two texts with the same label form a positive pair, and any text with a different label is a negative.

BatchAllTripletLoss averages every triplet in the batch that still breaks the margin.
BatchHardTripletLoss keeps only the hardest positive and the hardest negative for each anchor. It gives a strong signal but is sensitive to mislabelled rows.
BatchSemiHardTripletLoss picks negatives that are close but not closer than the positive, a middle ground.
BatchHardSoftMarginTripletLoss replaces the margin with a smooth penalty, so there is one fewer hyperparameter.

All of them need at least two texts per label in a batch, or there are no positives. Use BatchSamplers.GROUP_BY_LABEL.

The playground#

Pick a dataset and a loss, press Train, and watch four things:

Embedding space: each document is a square, training texts are dots and held-out texts are rings. A good fine-tune pulls each colour into its own cluster, including the rings the model never trained on.
Inside one batch: the same fixed batch, re-scored after every epoch, drawn the way that loss sees it.
Loss per epoch: training loss and the same loss on held-out texts.
Held-out quality: accuracy, how far apart right and wrong documents are, and how well cosine tracks the graded scores.

Dataset

Loss (the data format comes with it)

One batch of training rows:

Epochs Learning rate Batch size Scale Margin No-duplicates batches

Embedding space 32 dimensions projected to 2

Inside one batch

Loss per epoch train held-out

Held-out quality base model → fine-tuned

Try a new query against the six documents

Base model

Fine-tuned

The same run in sentence-transformers

Four experiments worth running#

1. Let duplicates into an MNRL batch. Uncheck No-duplicates batches. With 18 rows and 6 documents, a batch of 6 almost always holds two questions for the same article, and each is now told to push away the other’s positive, which is also its own answer. The training loss plateaus well above zero instead of falling to it.

2. Shrink the triplet margin to zero. With margin = 0, any triplet that is already ordered correctly contributes nothing, so training stops almost immediately. Held-out accuracy drops and the right-vs-wrong gap is less than half of what a 0.8 margin gives.

3. Compare CosineSimilarityLoss and CoSENT on the review dataset. In the batch view, CosineSimilarity pulls every dot onto its target line; CoSENT only spreads the three groups apart. On this toy data CosineSimilarity even gets a higher score correlation, because the dummy scores (1.0 / 0.4 / 0.0) sit exactly on the cosine scale. Real annotations rarely do, and that is where CoSENT’s order-only objective pays off.

4. BatchHard vs. BatchAll on ticket routing. In the matrix view, BatchHard marks the two cells per row that get a gradient (○ and ✕). BatchAll uses every violating cell, so as training goes on its signal gets diluted by triplets that are almost satisfied. Both end up clustering the tickets; watch how differently the matrix fills in on the way there.

The other families, briefly#

These don’t fit the playground’s data formats, but you will meet them:

ContrastiveLoss / OnlineContrastiveLoss take (text, text, 0 or 1). Positives are pulled together; negatives are pushed apart until they are beyond a margin. The online variant only uses the hard positives and hard negatives in each batch.
SoftmaxLoss trains a classifier on top of pair embeddings, as in the original SBERT paper on NLI data. It is mostly superseded by MNRL on (premise, entailment) pairs.
Distillation (MSELoss, MarginMSELoss, DistillKLDivLoss) copies a teacher. MarginMSE, which matches a cross-encoder’s score gap between a positive and a negative, is one of the strongest ways to train a retriever if you have a good re-ranker.
Unsupervised (DenoisingAutoEncoderLoss, ContrastiveTensionLoss) needs only raw text. Useful for domain adaptation before you have labels.
Wrappers (MatryoshkaLoss, Matryoshka2dLoss, AdaptiveLayerLoss) take another loss and apply it at several embedding sizes or layers. See Matryoshka Representation Learning for how that lets you truncate vectors later.

The full catalog#

Every sentence-transformer loss, with the data it needs and when to reach for it. Search by name, data format or use case.

MultipleNegativesRankingLoss default pick Ranking (anchor, positive) pairs

The go-to loss for semantic search. Uses other in-batch samples as negatives, so you only need (anchor, positive) pairs. Bigger batch = more negatives = stronger signal.

Good for

Semantic search
FAQ matching
Document retrieval

Strengths

Simplest data requirement
In-batch negatives scale with batch size
Strong out-of-the-box performance

Watch out for

May learn false negatives from batch
Quality depends heavily on batch size

Labels: None · Key params: scale = 20.0 (softmax temperature), similarity_fct = cos_sim (or dot product)

CachedMultipleNegativesRankingLoss default pick Ranking (anchor, positive) pairs

Memory-efficient MNRL that uses gradient caching to simulate massive batch sizes (4000+) on limited GPU memory. ~20% slower per step but each step sees far more negatives.

Good for

Large-scale training (100k+ pairs)
Memory-constrained GPUs

Strengths

Effective batch of thousands
Same data format as MNRL
Better gradients per step

Watch out for

~20% slower per step
More complex memory management

Labels: None · Key params: mini_batch_size = 32 (GPU encoding batch), scale = 20.0

MultipleNegativesSymmetricRankingLoss Ranking (anchor, positive) pairs

Symmetric variant of MNRL that treats both directions equally: A->B and B->A. Useful when the similarity relationship is bidirectional rather than query->document.

Good for

Paraphrase detection
Duplicate detection
Symmetric similarity

Strengths

Better for bidirectional tasks
Same simple data format

Watch out for

Slower than vanilla MNRL
Not ideal for asymmetric search

Labels: None · Key params: scale = 20.0

CachedMultipleNegativesSymmetricRankingLoss Ranking (anchor, positive) pairs

Cached version of symmetric ranking loss. Combines gradient caching with bidirectional similarity optimization.

Good for

Large-scale symmetric similarity

Strengths

Huge effective batch
Bidirectional

Watch out for

Slower
Complex setup

Labels: None · Key params: mini_batch_size = 32

GISTEmbedLoss Ranking (anchor, positive) pairs

Guided In-Sample Triplets: uses a guide model to identify true negatives among in-batch samples, solving MNRL's false-negative problem.

Good for

High-precision retrieval
When false negatives hurt

Strengths

Eliminates false negatives
Better than vanilla MNRL

Watch out for

Needs guide model (slow)
Higher GPU memory

Labels: None (guide model provides signal) · Key params: guide = SentenceTransformer('model')

CachedGISTEmbedLoss Ranking (anchor, positive) pairs

Memory-efficient variant of GIST that caches guide model embeddings. Combines the false-negative filtering with gradient caching.

Good for

Large-scale guided training

Strengths

GIST quality at scale
Memory efficient

Watch out for

Complex pipeline
Two models in memory

Labels: None · Key params: mini_batch_size = 32

MegaBatchMarginLoss Ranking (anchor, positive) pairs

Margin-based ranking loss optimized for very large batches. Uses a margin between positive and most-similar negative within a mega-batch for hard negative mining.

Good for

Large batch training
Margin-based ranking

Strengths

Effective hard negative mining
Works at scale

Watch out for

Needs large batches
More memory

Labels: None · Key params: positive_margin = 0.8, negative_margin = 0.3

TripletLoss Triplet (anchor, positive, negative) triplets

Classic triplet loss: explicitly push anchor closer to positive and away from negative by a margin. Best when you have carefully curated hard negatives.

Good for

Ranking with hard negatives
Re-ranking pipelines

Strengths

Explicit control over negatives
Works well with hard negatives
Interpretable margin

Watch out for

Need to mine/curate negatives
3x more data annotation effort
Margin tuning required

Labels: None (implicit in triplet structure) · Key params: distance_metric = COSINE, triplet_margin = 0.5 (tune lower if can't converge)

BatchAllTripletLoss Triplet Single sentences + class labels

Generates all valid triplets from class-labeled inputs within a batch. No explicit triplet construction needed -- just label your data.

Good for

Classification-based clustering
When you have class labels, not pairs

Strengths

Automatic triplet mining
Uses all valid combinations
Simple label-based data

Watch out for

Many easy triplets dilute signal
Requires class labels

Labels: Class labels (int) · Key params: distance_metric = COSINE, margin = 5

BatchHardTripletLoss Triplet Single sentences + class labels

Selects only the hardest positive (furthest same-class) and hardest negative (closest different-class) within each batch. Much stronger training signal.

Good for

Hard negative mining without explicit curation

Strengths

Strongest signal per batch
No manual negative selection

Watch out for

Can be unstable early in training
Noisy with small batches

Labels: Class labels (int) · Key params: distance_metric = COSINE, margin = 5

BatchHardSoftMarginTripletLoss Triplet Single sentences + class labels

Like BatchHardTripletLoss but uses a soft margin (log-sum-exp) instead of a fixed margin. No margin hyperparameter to tune.

Good for

When margin tuning is impractical

Strengths

No margin to tune
Harder examples get more gradient

Watch out for

Can be less stable
Requires class labels

Labels: Class labels (int) · Key params: distance_metric = COSINE

BatchSemiHardTripletLoss Triplet Single sentences + class labels

Uses semi-hard negatives: closer than the positive but still on the correct side of the margin. Balances learning signal with stability.

Good for

Stable training with automatic mining

Strengths

More stable than full hard mining
Good learning signal

Watch out for

May plateau if batch too small
Requires class labels

Labels: Class labels (int) · Key params: distance_metric = COSINE, margin = 5

CoSENTLoss Scored similarity (sentence_A, sentence_B, score) triples

Optimizes cosine similarity to match human-annotated scores. Uses a ranking-based objective that's more robust than direct regression (unlike CosineSimilarityLoss).

Good for

Semantic textual similarity (STS)
Paraphrase scoring
Graded relevance

Strengths

Learns nuanced similarity
Robust ranking objective
Works with continuous scores

Watch out for

Needs scored annotations
Slower to converge than MNRL

Labels: Float similarity (0-1) · Key params: scale = 20.0

AnglELoss Scored similarity (sentence_A, sentence_B, score) triples

Angular-based loss that optimizes embedding angles rather than distances. A modern alternative to CoSENTLoss with better gradient properties on hard examples.

Good for

STS tasks
When CoSENT plateaus

Strengths

Better gradients than CoSENT
Angle-aware optimization

Watch out for

Needs scored data
Newer, less battle-tested

Labels: Float similarity (0-1) · Key params: scale = 20.0

CosineSimilarityLoss Scored similarity (sentence_A, sentence_B, score) triples

Direct regression: minimizes MSE between predicted cosine similarity and target score. Simple but less effective than CoSENT/AnglE for most tasks.

Good for

Quick baselines
Simple similarity tasks

Strengths

Very simple
Easy to understand

Watch out for

Weaker than CoSENT/AnglE
Sensitive to score distribution

Labels: Float similarity (0-1) · Key params: loss_fct = nn.MSELoss()

ContrastiveLoss Contrastive (sentence_A, sentence_B, label) triples

Pulls similar pairs together and pushes dissimilar pairs apart. Requires binary labels indicating whether each pair is similar or dissimilar.

Good for

Binary similarity judgments
Verification tasks

Strengths

Works with binary labels
Intuitive objective

Watch out for

Needs both positive and negative pairs
No in-batch negatives

Labels: Binary (0 or 1) · Key params: distance_metric = COSINE, margin = 0.5

OnlineContrastiveLoss Contrastive (sentence_A, sentence_B, label) triples

Contrastive loss that dynamically focuses on hard positives and hard negatives within each batch, ignoring easy examples for a stronger signal.

Good for

Hard-example mining with binary labels

Strengths

Online hard mining
Stronger signal than vanilla contrastive

Watch out for

Still needs binary labels
Can be unstable

Labels: Binary (0 or 1) · Key params: distance_metric = COSINE, margin = 0.5

ContrastiveTensionLoss Contrastive Single sentences

Self-supervised contrastive learning from single sentences. Generates positive pairs by passing the same sentence through two independent encoders.

Good for

Unsupervised domain adaptation
No labeled data available

Strengths

No labels needed
Domain adaptation

Watch out for

Requires two encoders
Weaker than supervised losses

Labels: None

ContrastiveTensionLossInBatchNegatives Contrastive (anchor, anchor) duplicate pairs

Contrastive tension with in-batch negatives for stronger self-supervised learning. Uses (anchor, anchor) pairs and treats other batch items as negatives.

Good for

Self-supervised with better negatives

Strengths

Better than vanilla CT
Still no labels

Watch out for

Two encoders needed
Complex setup

Labels: None

SoftmaxLoss Classification (sentence_A, sentence_B) pairs + class label

Classic cross-entropy classification on top of [CLS] token or pooled embeddings. Maps pairs to discrete classes (entailment/contradiction/neutral for NLI).

Good for

NLI-based training
Multi-class similarity

Strengths

Standard classification setup
Works with NLI datasets

Watch out for

Adds classification head
Less direct for similarity

Labels: Class labels (int) · Key params: num_labels = 3 (for NLI)

MSELoss Distillation Sentences (teacher generates targets)

Minimizes mean squared error between student embeddings and teacher embeddings. Used for knowledge distillation from a larger model to a smaller one.

Good for

Model compression
Distilling large models

Strengths

Simple distillation
Flexible teacher choice

Watch out for

Needs teacher model
Bounded by teacher quality

Labels: Teacher embeddings (auto-generated)

MarginMSELoss Distillation (query, doc_pos, doc_neg) + teacher scores

Distillation loss that preserves relative score margins from a teacher (e.g., cross-encoder). Learns that the score gap between pairs should match the teacher's gaps.

Good for

Distilling cross-encoder rerankers into bi-encoders

Strengths

Preserves ranking margins
Strong for search

Watch out for

Complex data pipeline
Needs teacher scores

Labels: Teacher similarity scores · Key params: similarity_fct = pairwise_dot_score

DistillKLDivLoss Distillation (query, positive, negative) + teacher scores

KL-divergence distillation: matches the student's score distribution to the teacher's. More nuanced than MSE as it captures the full distribution shape.

Good for

High-fidelity distillation

Strengths

Captures distribution shape
Better than MSE for ranking

Watch out for

Complex setup
Needs teacher scores

Labels: Teacher score distributions

EmbedDistillLoss Distillation Sentences + teacher embeddings

Distills knowledge from teacher embeddings to student model with optional projection. Can distill across different embedding dimensions.

Good for

Cross-dimensional distillation

Strengths

Works across dimensions
Simple objective

Watch out for

Needs pre-computed teacher embeddings

Labels: Teacher embeddings

MatryoshkaLoss Wrapper Same as the wrapped loss

Not a standalone loss -- wraps any other loss to train embeddings that work at multiple dimensions. Truncate 384d to 64d and keep ~97% quality. Massive storage/speed savings.

Good for

Production deployment
Storage/speed optimization

Strengths

Near-free compression (3-6x)
Drop-in wrapper
No quality cliff

Watch out for

Slightly longer training
Must choose dimension list

Labels: Same as the wrapped loss · Key params: matryoshka_dims = [256, 128, 64, 32], matryoshka_weights = [1, 1, 1, 1]

Matryoshka2dLoss Wrapper Same as the wrapped loss

Two-dimensional Matryoshka: compresses both embedding dimensions and model layers. Enables early exit from transformer layers for even faster inference.

Good for

Maximum inference speed
Edge deployment

Strengths

Layer + dimension compression
Flexible speed/quality trade-off

Watch out for

Complex training
Needs layer-aware model

Labels: Same as the wrapped loss · Key params: matryoshka_dims = [256, 128, 64], n_layers_per_step = -1

AdaptiveLayerLoss Wrapper Same as the wrapped loss

Trains adaptive projection layers on top of each transformer layer, so you can get usable embeddings from any intermediate layer for faster inference.

Good for

Variable-speed inference
Layer-wise early exit

Strengths

Trade speed for quality at inference
Works with any base loss

Watch out for

More parameters to train
Moderate quality drop at shallow layers

Labels: Same as the wrapped loss · Key params: n_layers_per_step = -1

GlobalOrthogonalRegularizationLoss Wrapper Any (regularizes embeddings)

Regularization loss that encourages orthogonal embedding dimensions, reducing redundancy. Used as an auxiliary loss alongside a primary training objective.

Good for

Reducing dimension redundancy
Improving downstream compression

Strengths

Reduces dimension correlation
Pairs with any loss

Watch out for

Auxiliary only
May hurt if over-weighted

Labels: None · Key params: weight = balance with primary loss

DenoisingAutoEncoderLoss Unsupervised Single sentences (or damaged, original pairs)

Unsupervised: corrupts input sentences (deletion, swapping, insertion) and trains the model to reconstruct the original. Great for domain-adaptive pretraining.

Good for

Domain adaptation
Pretraining on unlabeled corpus

Strengths

No labels at all
Good for domain shift

Watch out for

Weak signal per example
Needs large corpus

Labels: None · Key params: decoder_name_or_path = model name, noise_fn = default (delete/swap/insert)

A short decision list#

Questions and passages? MNRL, the biggest batch you can fit, NO_DUPLICATES. Move to CachedMNRL when memory becomes the limit.
Can you mine hard negatives? Do it (sentence_transformers.util.mine_hard_negatives) and pass them as a third column to MNRL.
Graded human scores? CoSENT or AnglE. Use CosineSimilarityLoss only as a baseline.
Only class labels? BatchHardTriplet or BatchSemiHardTriplet with GROUP_BY_LABEL.
Have a strong re-ranker? Distil it into your retriever with MarginMSE.
Storage or latency matters? Wrap whichever loss you chose in MatryoshkaLoss.

The playground’s code panel shows the matching sentence-transformers script for whatever you last trained, so you can go from a setting that behaves well here to a real run on your own data.