В частности, следующий блок кода сравнивает использование nn.CrossEntropyLoss(reduction='mean') с loss_fn = nn.CrossEntropyLoss(reduction='none'), за которым следует loss.mean() . Результаты на удивление разные.
import torch
import torch.nn as nn
# Generate random predictions and labels
preds = torch.randn(8, 10, 100)
labels = torch.randint(high=100, size=(8, 10))
# replace some values with -100
labels[torch.rand(labels.shape) < 0.2] = -100
preds, labels = preds.view(-1, 100), labels.view(-1)
def compare_losses(preds, labels):
# Define loss functions
loss_fn = nn.CrossEntropyLoss(reduction='none')
loss_fn_mn = nn.CrossEntropyLoss(reduction='mean')
# Compute losses
losses = loss_fn(preds, labels)
weighted_loss = losses.mean()
# Compute mean loss using built-in mean reduction
loss = loss_fn_mn(preds, labels)
# Print and check if the results are identical
return torch.isclose(loss, weighted_loss), loss.item(), weighted_loss.item()
compare_losses(preds, labels)
Возвраты
(tensor(False), 4.997840404510498, 3.748380184173584)
Подробнее здесь: https://stackoverflow.com/questions/787 ... ction-mean
