Я создаю приложение, использующее вывод llm, поэтому попробовал vllm.
Поскольку для этого практически нет информации Библиотека, я хотел бы обратиться за помощью к экспертам.
Я использую модуль ROCm для аппаратного ускорения.
Фрагмент кода, который я приложил ниже, просто для проверки работы vllm в моей системе.
Вот код:
from vllm import LLM, SamplingParams
prompts = [
"Hello, my name is",
"The president of the United States is",
"The capital of France is",
"The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)
llm = LLM(model="facebook/opt-125m")
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
prompt = output.prompt
generated_text = output.outputs[0].text
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
Обратная связь:
WARNING 09-28 10:28:51 _custom_ops.py:18] Failed to import from vllm._C with ImportError('libcuda.so.1: cannot open shared object file: No such file or directory')
WARNING 09-28 10:28:57 config.py:1656] Casting torch.float16 to torch.bfloat16.
WARNING 09-28 10:28:57 config.py:376] Async output processing is only supported for CUDA or TPU. Disabling it for other platforms.
WARNING 09-28 10:28:57 config.py:681] Possibly too large swap space. 4.00 GiB out of the 7.09 GiB total CPU memory is allocated for the swap space.
INFO 09-28 10:28:57 llm_engine.py:226] Initializing an LLM engine (v0.6.1.dev238+ge2c6e0a82) with config: model='facebook/opt-125m', speculative_config=None, tokenizer='facebook/opt-125m', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=2048, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cpu, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=facebook/opt-125m, use_v2_block_manager=False, num_scheduler_steps=1, multi_step_stream_outputs=False, enable_prefix_caching=False, use_async_output_proc=False, use_cached_outputs=False, mm_processor_kwargs=None)
/home/dharsann/Documents/llm/.venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py
FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be deprecated in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transfor ... sues/31884warnings.warn(
WARNING 09-28 10:28:59 cpu_executor.py:328] CUDA graph is not supported on CPU, fallback to the eager mode.
WARNING 09-28 10:28:59 cpu_executor.py:354] Environment variable VLLM_CPU_KVCACHE_SPACE (GB) for CPU backend is not set, using 4 by default.
INFO 09-28 10:28:59 selector.py:217] Cannot use FlashAttention-2 backend for Volta and Turing GPUs.
INFO 09-28 10:28:59 selector.py:116] Using XFormers backend.
/home/dharsann/Documents/llm/.venv/lib/python3.12/site-packages/xformers/ops/fmha/flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
@torch.library.impl_abstract("xformers_flash::flash_fwd")
/home/dharsann/Documents/llm/.venv/lib/python3.12/site-packages/xformers/ops/fmha/flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
@torch.library.impl_abstract("xformers_flash::flash_bwd")
INFO 09-28 10:28:59 selector.py:217] Cannot use FlashAttention-2 backend for Volta and Turing GPUs.
INFO 09-28 10:28:59 selector.py:116] Using XFormers backend.
INFO 09-28 10:29:00 weight_utils.py:242] Using model weights format ['*.bin']
Loading pt checkpoint shards: 0% Completed | 0/1 [00:00
Подробнее здесь: https://stackoverflow.com/questions/790 ... -is-prompt
