Transformers/trainer.py: объект 'int' не имеет атрибута 'mean' ⇐ Python

[Anonymous]

Я пытаюсь обучить Орфей-3b, используя модифицированную версию блокнота Unsloth.
Я уже создал набор данных. Отправил набор данных в HuggingFace Hub. Преобразовал набор данных в тензоры, токенизировал записи набора данных.
Добавил несколько отпечатков в код, чтобы все проверить, но безуспешно.
Я бот-разработчик Python, но немного знаю Python. Буду очень признателен за любую помощь.
Что бы я ни делал, блоки постоянно терпят неудачу:
from transformers import TrainingArguments, Trainer, DataCollatorForLanguageModeling
from unsloth import is_bfloat16_supported

print(processed_train_ds[0])

data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False)
def orpheus_data_collator(features):
batch = {
"input_ids": torch.stack([f["input_ids"] for f in features]),
"attention_mask": torch.stack([f["attention_mask"] for f in features]),
"labels": torch.stack([f["labels"] for f in features]),
}
return batch

trainer = Trainer(
model = model,
train_dataset = processed_train_ds,
data_collator = data_collator, #eval_dataset = processed_eval_ds,
args = TrainingArguments(
per_device_train_batch_size = 8, # 16 fine for orpheus for 1024 seq length on 40 GB VRAM. 8 for 2048
gradient_accumulation_steps = 4,
warmup_steps = 5,
# max_steps = 60,
num_train_epochs = 3,
#eval_strategy = "steps",
#eval_steps = 0.2,
learning_rate = 2e-4, # try 2e-5 for FFT, 2e-4 for LoRA
fp16 = not is_bfloat16_supported(),
bf16 = is_bfloat16_supported(),
logging_steps = 1,
optim = "adamw_8bit",
weight_decay = 0.01, # Turn this on if overfitting
lr_scheduler_type = "constant",
seed = 3407,
output_dir = "outputs",
report_to = "none", # Use this for WandB or Tensorboard.
logging_dir = f"logs/{run_name}"
)
)
x = processed_train_ds[0]
print(x["labels"].unique())
print(x["labels"].dtype)
trainer_stats = trainer.train()

Ошибка:
tensor([ -100, 11, 13, ..., 156916, 156924, 156927])
torch.int64

==((====))== Unsloth - 2x faster free finetuning | Num GPUs used = 1
\\ /| Num examples = 7,064 | Num Epochs = 3 | Total steps = 333
O^O/ \_/ \ Batch size per device = 16 | Gradient accumulation steps = 4
\ / Data Parallel GPUs = 1 | Total batch size (16 x 4 x 1) = 64
"-____-" Trainable parameters = 97,255,424 of 3,398,122,496 (2.86% trained)

---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
Cell In[58], line 4
2 print(x["labels"].unique())
3 print(x["labels"].dtype)
----> 4 trainer_stats = trainer.train()

File ~/anaconda3/envs/py312/lib/python3.12/site-packages/transformers/trainer.py:2325, in Trainer.train(self, resume_from_checkpoint, trial, ignore_keys_for_eval, **kwargs)
2323 hf_hub_utils.enable_progress_bars()
2324 else:
-> 2325 return inner_training_loop(
2326 args=args,
2327 resume_from_checkpoint=resume_from_checkpoint,
2328 trial=trial,
2329 ignore_keys_for_eval=ignore_keys_for_eval,
2330 )

File :330, in _fast_inner_training_loop(self, batch_size, args, resume_from_checkpoint, trial, ignore_keys_for_eval)

File :71, in _unsloth_training_step(***failed resolving arguments***)

AttributeError: 'int' object has no attribute 'mean'


Подробнее здесь: https://stackoverflow.com/questions/798 ... ibute-mean