Огромная потеря тренировок при тонкой настройке Deberta-V3-Small с тренером Huggingface + Lora ⇐ Python

[Anonymous]

Я пытаюсь точно настроить Microsoft/deberta-v3-small с адаптерами Trainer + Peft Lora для задачи бинарной классификации (истина против транскриптов Lie).
Мой набор данных из базы данных MU3D. Это маленький (~ 320 образцов), а этикетки - 0/1 с 1 = истиной и 0 = ложь.
Вот пример формата моего набора данных, который я уже предварительно обработал и разделен на поезде, валидацию и тестовые наборы. class="s-table-container">


VideoID
SpeakerID
Veracity
Transcription
text_clean

< /thead>


bf003_1pt < /td>
bf003 < /td>
1 < /td>
Это образец текст < /td>
. Это образец. Это образец. /> < /tr>

bf003_2nl < /td>
bf003 < /td>
0 < /td>
это еще один тест < /td>
Это еще один тест < /td>
< /td>
. /> < /table> < /div>
i предварительно обработал текст с токенизатором (max_len = 512). Обучение проходит на NVIDIA RTX 2050 (4 ГБ VRAM). < /P>
Вот мой трубопровод < /p>
import os, numpy as np, pandas as pd, torch
from datasets import Dataset, DatasetDict
from sklearn.metrics import accuracy_score, precision_recall_fscore_support, average_precision_score, confusion_matrix
from transformers import (AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments,
Trainer, EarlyStoppingCallback, BitsAndBytesConfig)
from peft import LoraConfig, get_peft_model, TaskType
from pathlib import Path

SEED = 42
MODEL_NAME = "microsoft/deberta-v3-small" # fallback below if OOM
MAX_LEN = 192
OUTPUT_DIR = "artifacts_lora_deberta_small"

torch.manual_seed(SEED); np.random.seed(SEED)

# ---------- IO ----------
train = pd.read_csv("train_subject_disjoint.csv")
val = pd.read_csv("val_subject_disjoint.csv")
test = pd.read_csv("test_subject_disjoint.csv")

def to_hfds(df):
return Dataset.from_pandas(
df[["text_clean","Veracity","VideoID","SpeakerID"]]
.rename(columns={"text_clean":"text","Veracity":"label"}),
preserve_index=False
)

ds = DatasetDict(train=to_hfds(train), validation=to_hfds(val), test=to_hfds(test))

< /code>
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)
def tok(batch):
return tokenizer(batch["text"], max_length=MAX_LEN, truncation=True, padding="max_length")
ds = ds.map(tok, batched=True)
ds = ds.remove_columns(["text","VideoID","SpeakerID"])
ds.set_format(type="torch")

base = AutoModelForSequenceClassification.from_pretrained(
"microsoft/deberta-v3-small" ,
num_labels=2,
device_map="cuda")

lora_cfg = LoraConfig(
task_type=TaskType.SEQ_CLS,
r=16,
lora_alpha=32,
lora_dropout=0.1,
target_modules=["query_proj","key_proj","value_proj","dense"]
)
model = get_peft_model(base, lora_cfg)
model.print_trainable_parameters()

args = TrainingArguments(
output_dir=OUTPUT_DIR,
eval_strategy="epoch",
save_strategy="epoch",
load_best_model_at_end=True,
metric_for_best_model="f1_macro",
greater_is_better=True,

num_train_epochs=50,
per_device_train_batch_size=4, # small to fit 4GB
per_device_eval_batch_size=16,
gradient_accumulation_steps=4, # effective train batch = 16
fp16= True, # mixed precision
bf16=False, # set True if your GPU supports bf16
learning_rate=2e-5,
weight_decay=0.01,
warmup_ratio=0.1,

logging_steps=50,
save_total_limit=2,
seed=SEED,

gradient_checkpointing=True, # big memory saver on 4GB
)

trainer = Trainer(
model=model,
args=args,
train_dataset=ds["train"],
eval_dataset=ds["validation"],
tokenizer=tokenizer,
compute_metrics=compute_metrics,
callbacks=[EarlyStoppingCallback(early_stopping_patience=10)]
)

trainer.train()
print("\n[VAL]", trainer.evaluate())

< /code>
However, when my training loop was running, I observed a huge training loss below



Epoch
Training Loss
Validation Loss
Accuracy
F1 Macro
F1 Weighted
Pr Auc
Precision Macro
Recall Macro




1
No log
0.697312
0.500000
0.333333
0.333333
0.532561
0.250000
0.500000


2
No log
0.697031
0.500000
0.333333
0.333333
0.534184
0.250000
0.500000


3
No log
0.696202
0.500000
0.333333
0.333333
0.532561
0.250000
0.500000


4
0.700200
0.695277
0.500000
0.333333
0.333333
0.532101
0.250000
0.500000


5
0.700200
0.694190
0.500000
0.333333
0.333333
0.534942
0.250000
0.500000


6
0.700200
0.693882
0.500000
0.333333
0.333333
0.534942
0.250000
0.500000


7
0.694800
0.693520
0.500000
0.333333
0.333333
0.533724
0.250000
0.500000


8
0.694800
0.693351
0.500000
0.333333
0.333333
0.532101
0.250000
0.500000


9
0.694800
0.693283
0.500000
0.333333
0.333333
0.532101
0.250000
0.500000


10
0.697500
0.693154
0.500000
0.333333
0.333333
0.533724
0.250000
0.500000


11
0.697500
0.693144
0.500000
0.333333
0.333333
0.529720
0.250000
0.500000



I was just wondering, how can I improve the training loss? I am kind of new to fine-tuning models. Would I swap this model out with a new one?
Thanks

Подробнее здесь: https://stackoverflow.com/questions/797 ... ce-trainer